Getting disconnects after 3000 users on single game server.

[We are using Loadbalancing solution provided from Photon.]

Our game servers are Intel Xeon E5 2686 with 64GB ram
We can easily reach 3000 users on our game servers. CPU load is 60% and ram usage is 10%. So i believe resource-wise we do not have any problems. But the moment we try to go over 3000 users our server starts to disconnect peers very slowly but steadily. 1 user every 20seconds or so.
When the disconnects start we have about
7500 messages in/sec
38000 messages out /sec

The most common log is " Timeout occurred: ClientTimeoutDisconnect" but we sometimes have "ManagedDisconnect" too. Other than that we have zero error log


Comments

  • hi, @emrahsungu

    please answer some question:
    what is your license limit?
    what do you players do: are they all in one big room, in many rooms? when you add new players do you add them to existing rooms or create new ?

    clientTimeoutDisconnect means that client was not able to send neither ping nor acks, this may mean that it gets so many incomming messages that should handle them all.

    what is your client when you do a tests? how do you test?

    best,
    ilya
  • emrahsungu
    edited August 2017
    @chvetsov. THanks for the answer. We have licence for the unlimited users. We are currently doing load tests ON GAME SERVERS before releasing our game. Since this is a load test, we are doing what would be heaviest in our use case. Each user creates one room but the total peer did not change (almost) for different cases such as 3~5 players per room (Max 5 players per room)

    We have written test client in C# windows.forms. Our clients are running on Amazon servers. A single Amazon Server creates about 800 clients so we are using 4-6 AWS instances. CPU usage is around 30% with 50%(total memory 16gb) of memory usage

    Extra info: In the first post,where disconnects started at 3000 peers, we used AWS instance with
    ※Intel Xeon E5 2686 with 64GB ram with 16 cores m4.4xlarge

    In the meantime we carried out another Load test on a better spec AWS with
    ※Intel Xeon E5 2676 with 160GB ram with 40 cores m4.10xlarge CPU USAGE around 50% Memory usage around 7%
    This time max peers without any disconnects is around 4500.

    [Master server almost is under zero stress. CPU usage is about 3%]

    Here is a simple simulation for what our players do:
    -----------------------------
    Master connect,
    Establish encryption,
    Authentication,
    Create One Room,
    Disconnect.
    -----------------------------
    Connect to game server.
    Establish encryption,
    Authentication,
    Create One Room,
    Start Battle
    Send attack operation every ~200/300 ms (Dictionary{{0,1L},{10,20L}}) Actual data
  • guys, i would ask you to do next simple thing.
    please register performance counters for photon and start collect them, then repeat you tests.
    zip results and send us to developer@photonengine.com

    best,
    ilya
  • @chvetsov, we have already registered performance counters which ones do you need? since we are already testing, we can swiftly create new ones
  • using photon control you may create logging set and start loggin. here you may see a screenshot https://doc.photonengine.com/en-us/onpremise/current/performance/photon-counters#_ppc_win, where you may find photon control menu.

    so, again you need to create logging set and then start it using photon control menu. counters will be collected to 'C:\PerfLogs\Admin' folder

    best,
    ilya
  • @chvetsov we have sent an メールemail to developer@photonengine.com with log
  • kennethharder
    edited January 2018
    Hey @emrahsungu, did you guys figure out what was happening? We experience something similar and are looking for a solution. See my post for details on our issue: http://forum.photonengine.com/discussion/11177/we-are-seeing-clienttimeoutdisconnects-in-bulk/
  • hi, @kennethharder

    something similar is not a description. please, give us all details which are related.
    do not share here your appId.

    you also may write to developer@photonengine.com if some info is private

    best,
    ilya
  • Hey @chvetsov, I have added a link in my previous comment.