How to stress-test server

quarion

Hello.
I am wondering about creating automatic stress-test for our game server. Do you have any advice?
Maybe I can simply create many PhotonClient's (from LiteLobby), each with one PhotonPeer in it - all in single application, single thread.
Or maybe it must be a little more sophisticated?
Best regards,
Filip

Tobias

We don't have proper guides to load testing, but we wanted to create one soon.
Let me outline a few things out of my head and from what we do have.

Testing Setup

• Use a low(er)-end machine to run the server. It's easier to stress it and create edge cases of load.
• Use several machines in your local network to simulate clients.
• Keep an eye on all of the involved machine's CPU and network. Either side might get overwhelmed.
• The test client (process) should be able to spawn a arbitrary number of connections (based on LitePeer or LoadBalancingClient or...) and make use of ThreadPools to keep them alive.
• Write the test client as easy as possible. Use threads, fake data, remove graphics and don't expect input unless necessary. Make sure each client has OK update rates.
• Use DotNet or Unity libraries above version 3.0.1.3. Check your lib's dll for the version and update.
• Gather Counters (described below) to check your server's performance.
• In terms of performance: Logging might ruin performance, so reduce it.

What to look for:

• At first, look at "obvious" counters, like: how is the CPU load, how many Peers / Connections Active, how many Commands / sec
• Then check if the server is busy, by comparing how many of the "active" I/O, Business and ENet Threads are currently "processing", if we have messages in the ENet Queue and so on...
• Then check how well the "outgoing" traffic is handled, like the amount of Commands Resent / sec (if clients don't send ACKs, Photon resends the command), the amount of Queued Reliable Commands (the amount of commands that are sent, but not yet acknowledged) and so on... if these values are high, it's very likely a client problem
• In Photon's "doc" folder, there is a photon-perfcounter.pdf where all counters are listed

We updated the client SDKs and libraries for Unity and DotNet recently. The new versions got a lot of performance improvements (especially for load testing).
In best case, you would also update to the recent server SDKs. We released Photon 3 RC9 and if that's far from your version, you want to trade upgrade-effort for stability.

How to collect counters

The order of these steps is important!
1.) stop photon, if it's running
2.) from photon control -> performance counters - install counters, - create logging set
3.) start "perfmon" from command line.. under "data collector sets" -> "user defined": choose the photon_perf_log in the RIGHT window pane -> properties -> set the sample interval to 1 second
(the default of 1 minute does not give us enough data here)
4.) start photon
5.) photon control -> performance counters -> start logging
6.) run load test
7.) photon control -> performance counters -> stop logging
8.) Get performance logs from C:\perflogs\admin

If counter log creation from Photon Control fails:
from \deploy\bin_tools\perfmon:
logman.exe create counter photon_perf_log -si 00:01 -v mmddhhmm -cf logman.config.txt

I hope this helps you a step further.

quarion

That's a lot of helpful info, thank you very much

Adlearee

I have been around internet forums trying to find a good guide about stress testing. Although people had some useful info, but you have listed the most useful stuff in a single post. Amazing. And thanks a million for such a great post.