Photon Cloud issues on 12/06 - 12/08
Options
![[Deleted User]](https://wa.vanillicon.com/v2/a2145153dca14f6987b6e846db7bacab.svg)
Hey everybody,
we had a troubled weekend. Most of you noticed ongoing issues with Photon Cloud: disconnects, high latency, failures during game creation and even downtimes. So, first of all, I want to apologize: for the technical issues as well as for the lack of communication. We were so busy to solve the technical issues that we missed several tickets, forum posts, etc. - sorry about that. I'll come back to all of you and answer your questions in the next hours.
What happened?
Photon Cloud has seen a hugh traffic boost over the last weeks - in some regions, we have 2-3 times the amount of users now, compared to 2 months ago. With success comes a challenge: we've constantly increased our capacities, added a new region and evaluated hosting providers for the last weeks.
However, our efforts were not enough this weekend. There was a delay in the delivery of new hardware, an unexpected switch of user traffic between different Photon Cloud regions, a significant increase in traffic in general, another lack in capacity caused by a server maintenance of a hosting provider and finally a nasty Photon bug that prevented us from handling thousands of simultaneous reconnects after a restart.
Pretty much to handle simultaneously! For the whole weekend, we were adding server capacity and tried to make sure that as many players as possible could play during their "peak times". In some cases, we needed to apply changes that introduced higher latency, and unfortunately we needed to take Photon Cloud offline in several regions to re-establish a stable environment (and to find & work around that bug mentioned above).
Full outages occurred:
- Photon Cloud EU: Saturday, 12/07, between ~ 8:30 and 11:30 AM UTC
- Photon Cloud US: Saturday, 12/07, between ~ 6 and 7 PM UTC
Game creation was severly impacted:
- Photon Cloud EU: Saturday, 12/07, between 5 and 6 PM UTC
- Photon Cloud US: Saturday, 12/07, between 4 and 7 PM UTC
- Photon Cloud ASIA: Sunday, 12/08, between 6 and 8 AM UTC
... in addition to many shorter "hickups" throughout the day.
Current status:
By now, all major issues are resolved and Photon Cloud is stable on all regions. Latency might be a bit higher and you might see a slightly higher disconnect rate at the moment - this will be resolved as well in the next days. We are paying close attention to all environments and will make sure that your users will have a flawless game experience with Photon Cloud.
What's next?
We have learned several important lessons this weekend, and it is our top priority to increase the stability & availability of Photon Cloud. We are taking these measurements:
- more capacity will be added during the next week - in production as well as "on standby", so that we can make sure to guarantee availability and best performance, no matter how much traffic you throw on us :-)
- we'll probably spread out our over different hosting providers to be more flexible in case of any issues
- and, most important: we will introduce a mechanism called "Name Server" that will allow us to redirect Photon Cloud clients to different servers or regions on the fly, without any client-side changes.
This will give us several benefits: we can prevent "flooding", add separate servers for customers with high traffic on the fly, replace faulting servers without any downtimes etc. - in summary, we will gain a great deal of stability, redundancy and flexibility for Photon Cloud, which is what we are lacking most at the moment. These features are in a "closed beta" stadium right now and we hope to release them into production before the end of this year.
Finally...
I would like to apologize for all the issues you have encountered during the weekend. I hope I could answer most questions and give enough background information to understand the situation.
Please let me know if you have any questions, I'll be happy to answer them and help out in case there might be any more issues.
Best regards,
Nicole
PS: for latest Photon Cloud status notifications, please follow us on twitter: http://twitter.com/ExitGamesStatus
we had a troubled weekend. Most of you noticed ongoing issues with Photon Cloud: disconnects, high latency, failures during game creation and even downtimes. So, first of all, I want to apologize: for the technical issues as well as for the lack of communication. We were so busy to solve the technical issues that we missed several tickets, forum posts, etc. - sorry about that. I'll come back to all of you and answer your questions in the next hours.
What happened?
Photon Cloud has seen a hugh traffic boost over the last weeks - in some regions, we have 2-3 times the amount of users now, compared to 2 months ago. With success comes a challenge: we've constantly increased our capacities, added a new region and evaluated hosting providers for the last weeks.
However, our efforts were not enough this weekend. There was a delay in the delivery of new hardware, an unexpected switch of user traffic between different Photon Cloud regions, a significant increase in traffic in general, another lack in capacity caused by a server maintenance of a hosting provider and finally a nasty Photon bug that prevented us from handling thousands of simultaneous reconnects after a restart.
Pretty much to handle simultaneously! For the whole weekend, we were adding server capacity and tried to make sure that as many players as possible could play during their "peak times". In some cases, we needed to apply changes that introduced higher latency, and unfortunately we needed to take Photon Cloud offline in several regions to re-establish a stable environment (and to find & work around that bug mentioned above).
Full outages occurred:
- Photon Cloud EU: Saturday, 12/07, between ~ 8:30 and 11:30 AM UTC
- Photon Cloud US: Saturday, 12/07, between ~ 6 and 7 PM UTC
Game creation was severly impacted:
- Photon Cloud EU: Saturday, 12/07, between 5 and 6 PM UTC
- Photon Cloud US: Saturday, 12/07, between 4 and 7 PM UTC
- Photon Cloud ASIA: Sunday, 12/08, between 6 and 8 AM UTC
... in addition to many shorter "hickups" throughout the day.
Current status:
By now, all major issues are resolved and Photon Cloud is stable on all regions. Latency might be a bit higher and you might see a slightly higher disconnect rate at the moment - this will be resolved as well in the next days. We are paying close attention to all environments and will make sure that your users will have a flawless game experience with Photon Cloud.
What's next?
We have learned several important lessons this weekend, and it is our top priority to increase the stability & availability of Photon Cloud. We are taking these measurements:
- more capacity will be added during the next week - in production as well as "on standby", so that we can make sure to guarantee availability and best performance, no matter how much traffic you throw on us :-)
- we'll probably spread out our over different hosting providers to be more flexible in case of any issues
- and, most important: we will introduce a mechanism called "Name Server" that will allow us to redirect Photon Cloud clients to different servers or regions on the fly, without any client-side changes.
This will give us several benefits: we can prevent "flooding", add separate servers for customers with high traffic on the fly, replace faulting servers without any downtimes etc. - in summary, we will gain a great deal of stability, redundancy and flexibility for Photon Cloud, which is what we are lacking most at the moment. These features are in a "closed beta" stadium right now and we hope to release them into production before the end of this year.
Finally...
I would like to apologize for all the issues you have encountered during the weekend. I hope I could answer most questions and give enough background information to understand the situation.
Please let me know if you have any questions, I'll be happy to answer them and help out in case there might be any more issues.
Best regards,
Nicole
PS: for latest Photon Cloud status notifications, please follow us on twitter: http://twitter.com/ExitGamesStatus
0
Comments
-
Huh - as soon as I posted this, we encountered another outage on Photon Cloud US, 12/08 from approximately 3:30 - 4:15 PM UTC.
This is now resolved as well.0
This discussion has been closed.