Disconnects - hundreds at a time

LudemeGames

This is a graph of the session count for my load tests. At periodic intervals, clients disconnect hundreds at a time for no apparent reason and then things continue on steadily, and then it happens again.

The load test itself is pretty basic, start up a bunch of clients, they start up some games and periodically make moves in the games. The clients and server are running on the same machine.

What do you recommend for diagnosing this issue? The only thing I could think of is that perhaps a gc is running here, and that period if no communication causes problems, but I'm a bit skeptical of that answer and not sure how to go about testing the theory or correcting that issue.

LudemeGames

It may be worth noting that no matter how many clients I add (500, 1000, ?) the first "purge" happens right after the ramp up in clients. I don't think this has anything to do with client behavior, since clients get busy right after they have connected and don't wait for other clients to connect.

It should be noted that the blank areas in the graph between the large ramp, the tall plateau, and short plateau do not have 0 clients. For whatever reason, the server does not draw into any of the performance graphs while it is purging.

Kaiserludi

Hi LudemeGames.

Your setup looks seriously flawed to me.
"The clients and server are running on the same machine." This is an absolute No Go for load tests and makes the results totally useless.

You should run the server on one machine and use as many other machines for the the clients as needed to not make the client side the bottle neck. Only that way you can see at which number of clients you really run into server side limitations.
With your current setup it's very likely, that the number of clients that run on that machine is just higher than what that machine can handle without issues. Probably your connections just time out because your machine doesn't have enough processing power to handle all those clients + the server side in time.

[Deleted User]

I agree with Kaiserludi.

You need to move the clients to a different machine - and don't run more than, say, 200-300 clients per machine (depending on the specs and the amount of load you create). Photon server is optimised to handle hundreds of connections on one machine; but our client libraries are not...

Loadtests are pretty hard to do "right", and we would need to know lots more details to give detailed advice, but this is very likely the root cause for your issues.

LudemeGames

I don't have many machines to test with, but thanks I'll try and see what I can round up to do the test.

LudemeGames

Nicole wrote:

Loadtests are pretty hard to do "right"

Is there documentation somewhere on doing them right to avoid other pitfalls? If not, I imagine this would be a good candidate to add to the standard documentation.

[Deleted User]

Unfortunately, ther's no standard documentation about load tests yet - but we definitely plan to add it!