Photon Server Self Shutdown Issue

Hi, I've had no issues with our game server until recently. We've had 3 server self shutdowns in the past 2 weeks and I'm not sure what is causing the shutdowns.

Here is the basic configuration:


I have an unlimited CCU license
Our peak CCU around the time of the crashes is about ~900.

Here is the server log prior to the shutdown:


2019-03-25 10:10:50,648 [70] ERROR Debug - [ERROR] Facebook.FacebookOAuthException: (OAuthException - #190) Error validating access token: The session has been invalidated because the user changed their password or Facebook has changed the session for security reasons.
at Facebook.FacebookClient.ProcessResponse(HttpHelper httpHelper, String responseString, Type resultType, Boolean containsEtag, IList`1 batchEtags)
at Facebook.FacebookClient.Api(HttpMethod httpMethod, String path, Object parameters, Type resultType)
at Facebook.FacebookClient.Get(String path, Object parameters, Type resultType)
2019-03-25 10:10:52,121 [51] ERROR Debug - [ERROR] Facebook.FacebookOAuthException: (OAuthException - #190) Error validating access token: The session has been invalidated because the user changed their password or Facebook has changed the session for security reasons.
at Facebook.FacebookClient.ProcessResponse(HttpHelper httpHelper, String responseString, Type resultType, Boolean containsEtag, IList`1 batchEtags)
at Facebook.FacebookClient.Api(HttpMethod httpMethod, String path, Object parameters, Type resultType)
at Facebook.FacebookClient.Get(String path, Object parameters, Type resultType)
at Server.Facebook.FacebookSystem.GetBasicInfoByActionToken(String accessToken) in F:\GitHub\PitayaGames-Danh-Bai\Backend\src-server\Photon\Server\Facebook\FacebookSystem.cs:line 16
2019-03-25 10:11:12,538 [70] ERROR Debug - [ERROR] Facebook.FacebookOAuthException: (OAuthException - #190) Error validating access token: The session has been invalidated because the user changed their password or Facebook has changed the session for security reasons.
at Facebook.FacebookClient.ProcessResponse(HttpHelper httpHelper, String responseString, Type resultType, Boolean containsEtag, IList`1 batchEtags)
at Facebook.FacebookClient.Api(HttpMethod httpMethod, String path, Object parameters, Type resultType)
at Facebook.FacebookClient.Get(String path, Object parameters, Type resultType)
at Server.Facebook.FacebookSystem.GetBasicInfoByActionToken(String accessToken) in F:\GitHub\PitayaGames-Danh-Bai\Backend\src-server\Photon\Server\Facebook\FacebookSystem.cs:line 16
2019-03-25 10:11:13,991 [33] ERROR Debug - [ERROR] Facebook.FacebookOAuthException: (OAuthException - #190) Error validating access token: The session has been invalidated because the user changed their password or Facebook has changed the session for security reasons.
at Facebook.FacebookClient.ProcessResponse(HttpHelper httpHelper, String responseString, Type resultType, Boolean containsEtag, IList`1 batchEtags)
at Facebook.FacebookClient.Api(HttpMethod httpMethod, String path, Object parameters, Type resultType)
at Facebook.FacebookClient.Get(String path, Object parameters, Type resultType)
at Server.Facebook.FacebookSystem.GetBasicInfoByActionToken(String accessToken) in F:\GitHub\PitayaGames-Danh-Bai\Backend\src-server\Photon\Server\Facebook\FacebookSystem.cs:line 16
2019-03-25 10:11:14,641 [27] ERROR Debug - [ERROR] Facebook.FacebookOAuthException: (OAuthException - #190) Error validating access token: The session has been invalidated because the user changed their password or Facebook has changed the session for security reasons.
at Facebook.FacebookClient.ProcessResponse(HttpHelper httpHelper, String responseString, Type resultType, Boolean containsEtag, IList`1 batchEtags)
at Facebook.FacebookClient.Api(HttpMethod httpMethod, String path, Object parameters, Type resultType)
at Facebook.FacebookClient.Get(String path, Object parameters, Type resultType)
at Server.Facebook.FacebookSystem.GetBasicInfoByActionToken(String accessToken) in F:\GitHub\PitayaGames-Danh-Bai\Backend\src-server\Photon\Server\Facebook\FacebookSystem.cs:line 16
2019-03-25 10:11:15,097 [79] ERROR Debug - [ERROR] Facebook.FacebookOAuthException: (OAuthException - #190) Error validating access token: The session has been invalidated because the user changed their password or Facebook has changed the session for security reasons.
at Facebook.FacebookClient.ProcessResponse(HttpHelper httpHelper, String responseString, Type resultType, Boolean containsEtag, IList`1 batchEtags)
at Facebook.FacebookClient.Api(HttpMethod httpMethod, String path, Object parameters, Type resultType)
at Facebook.FacebookClient.Get(String path, Object parameters, Type resultType)
at Server.Facebook.FacebookSystem.GetBasicInfoByActionToken(String accessToken) in F:\GitHub\PitayaGames-Danh-Bai\Backend\src-server\Photon\Server\Facebook\FacebookSystem.cs:line 16
2019-03-25 10:12:40,426 [67] ERROR Debug - [ERROR] OperationResponse: Disconnected, Code 20, UserId
2019-03-25 10:13:39,408 [68] ERROR Debug - [ERROR] OperationResponse: Disconnected, Code 21, UserId
2019-03-25 10:18:35,164 [45] ERROR ExitGames.Threading.FailSafeBatchExecutor - System.InvalidOperationException: Collection was modified; enumeration operation may not execute.
at System.ThrowHelper.ThrowInvalidOperationException(ExceptionResource resource)
at System.Collections.Generic.Dictionary`2.ValueCollection.Enumerator.MoveNext()
at System.Linq.Enumerable.WhereEnumerableIterator`1.MoveNext()
at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
at Server.RoomSystem.ServerLobby.GetRoomInfo(Int32 skip, Int32 limit, Int64 minCoin, Int64 maxCoin, Boolean isVisible) in F:\GitHub\PitayaGames-Danh-Bai\Backend\src-server\Photon\Server\RoomSystem\ServerLobby.cs:line 62
at Server.MasterServer.MasterClient.HandleGetListRoomOperation(GetListRoomOperationRequest getListRoomOperationRequest) in F:\GitHub\PitayaGames-Danh-Bai\Backend\src-server\Photon\Server\MasterServer\MasterClient.cs:line 3784
at Server.MasterServer.MasterClient.HandleGetListRoomOperation(OperationRequest operationRequest) in F:\GitHub\PitayaGames-Danh-Bai\Backend\src-server\Photon\Server\MasterServer\MasterClient.cs:line 3771
at Server.MasterServer.MasterClient.OnOperationRequestAsync(OperationRequest operationRequest, SendParameters sendParameters) in F:\GitHub\PitayaGames-Danh-Bai\Backend\src-server\Photon\Server\MasterServer\MasterClient.cs:line 109
at ExitGames.Threading.FailSafeBatchExecutor.Execute(Action action) in c:\(Work)\(EG)\exitgames-libs\src\Core\Threading\FailSafeBatchExecutor.cs:line 111
2019-03-25 10:20:36,469 [1] INFO Photon.SocketServer.ApplicationBase - Application is stopping: AppId=Server
2019-03-25 10:20:36,543 [9] ERROR Debug - [ERROR] Event: Disconnected, Code 2, UserId c5890dc0-7045-4124-9c75-0e70c6e458da
2019-03-25 10:20:36,564 [1] INFO Debug - [INFO] Server had shutdown
2019-03-25 10:20:36,572 [1] INFO Photon.SocketServer.ApplicationBase - Application stop: AppId=Server

If there is any other information that I can provide to gain some insight on this issue, please let me know.

Thank you for your time!

Comments

  • Forgot to include this information:

    MaxMessageSize="1024000"
    MaxQueuedDataPerPeer="1024000"
    PerPeerMaxReliableDataInTransit="512000"
    PerPeerTransmitRateLimitKBSec="512"
    PerPeerTransmitRatePeriodMilliseconds="200"
    MinimumTimeout="5000"
    MaximumTimeout="60000"
    DisplayName="Server Danh Bai"
  • hi, @trungmmm

    this could happen if you have stack overflow, or you have updated some files in server folder.
    Please provide us full logs from bin_Win64/logs

    best,
    ilya