Frequent disconnect on local network

Pelekas

Howdy all,

I'm hoping to find some clues for my disconnect problems.

Setup:
I have 2 PC's running on a local network. One is running a custom photon server app (authentication server), the other is running a dotnet client peer.
I have quite a lot of small network packages.
Some stats for this test:
* messages in: 30/sec
* messages out: 100/sec
* bytes in: 15k/sec
* bytes out: 25k/sec
These are mostly unreliable messages (only 1 or 2 reliable messages are send/sec).

Results:
When I run both server and client on the same PC, I have no problems.
When I run them on 2 separate PC's, I have frequent and completely unpredictable disconnects. Frequency of disconnects is on average about 1 every 30 seconds.

When I monitor the network traffic between client and server, I notice some periodic drops = sometimes the network traffic drops to zero for 1 second. It seems this causes the problems. Sometimes the photon client doesn't mind the network drop, sometimes it will give me a disconnect.
I have no idea what is causing this network drop, and frankly I don't really care, as the end game should be able to survive far worst network behavior.

I'm using more or less the standard photon server settings.
Which settings should I start tweaking to survive this connection drop? (Or am I doing something else horribly wrong?)

Tobias

Which side causes the traffic drop to 0?
You might want to check if the clients framerate is constant and does not "stall" suddenly. If you load assets, Unity won't call the Update methods and thus the client can't send anything.

This could be fixed by using the isMessageQueueRunning property:
http://doc.exitgames.com/photon-cloud/P ... #_messageQ

Pelekas

Thanks for the response Tobias.

I'm using the dotnet SDK, not PUN, so asset loading & isMessageQueueRunning is not an issue.
But, you made me think more about looking at the client instead of the server.

First of all, the traffic drops are indeed client side, not server side. Even with the simplest of clients (= not much more then an infinite loop with a Thread.Sleep(25) and Peer.Service()), I still have the traffic drops.
So, I started tweaking the client peer properties.

Raising the Peer.SentCountAllowance did the trick.
=> I did some long running and traffic intensive tests, and upping this property stopped the disconnects.
What would be a reasonable value for Peer.SentCountAllowance in this kind a scenario? Could upping it too much cause some other problems?

I still don't know what is causing the traffic drops, and I'm starting to get curious about these. I'll be doing some more tests the next couple of days to find the exact problem.

In the mean time, thanks for pushing me in the right direction.

Tobias

The reliable udp protocol we use is geared towards sending many smaller messages instead of fewer big ones. We didn't get around to optimizing it for larger messages yet.
A Peer.SentCountAllowance of 5 should usually be fine but make it 10 or 15 or so. What will happen is that the client is allowed to repeat messages more often, if their ACKs don't arrive in time. This means you clients accept longer delays but keep the connection for a longer time. In the end, the PhotonPeer.DisconnectTimeout will trigger after 10 seconds without ack (measured per command) but that's quite a long time.

Pelekas

Funny... and there I was optimizing my code with bulk packages, instead of many smaller ones ;o)
What would be an ideal packet size?

Tobias

The packages are limited to 1200 bytes payload. Considering you get some overhead per message and for the type-info, maybe stay below 1kB. Small messages are aggregated if they fit into the same package, so doing smaller updates is also fine.
Chunks that break the 1200 bytes barrier will get fragmented into multiple reliable commands and packages. This is another reason why bigger messages are not always better.

Pelekas

Hello guys,

I did a lot of testing the last week, and I think I found the real problem.
But I would like some advice on how to handle it.

First of all: my tests and my results.
First test: a clean photon server install + a very simple photonPeerListener connected to a LiteApp.
The client peer generated a constant network traffic at intervals of 1 event/50ms, /100ms or /200ms. At all intervals, the results are the same.
Result: every minute (exactly every 60 seconds), my incoming bytes drop to zero for about 1 second. This happens on the server as well as on the client.

This is when I started to realize that the problem is not photon related.

Further tests include doing the same with long downloads and other constant network packet sources (for instance the firefall beta which gives a nice visual indication of network problems).
The metric that I was watching = in PerfMon the "Network Interface: Bytes Received/sec".
All tests indicate the same drop, every 60 secs.

When I asked around, somebody mentioned that this is a common problem with wifi routers.
Yes, my photon server and client are both on a wifi LAN connection. When I filter out the network traffic, I noticed a 4k heartbeat every 60secs just before the connection drop.
The heartbeat occurs on wifi and on ethernet connections, but the connection drop only occurs on wifi. LAN keeps a stable network.

So: ultimate testsetup
=> I put my photon server on an ethernet cable and my client on wifi
=> bingo: we have a stable client/server connection, even with all default settings.

Questions:
* I know having a server on a wifi is not the best of ideas, but are there any settings that I could tweak to get a stable connection?
* Do you guys know about this problem with wifi routers, and could you suggest a solution. Even when this means upgrading my router to a new model (currently, I'm using a linksys WRT160N, which is already over 5 years old). But I would like to have some references, cause I can't find any relevant links on google.
* I could ofcourse start using the photon cloud, but is it possible to have my own custom photon apps uploaded? Do you have some resource references on that topic?


Anyway, thanks for the help!

[Deleted User]

Wow - good catch. And a big "thank you" for coming back and posting your results!

From all I know, we've not experienced that kind of problem before.
4k seems really big for a simple "heartbeat" frame, and it should certainly not bring your network down. Unfortunately, I can not give any real advice - besides "try to update your drivers and firmware, or try to borrow another router for a quick test run.". You might have more luck at a dedicated hardware / wifi forum.

Photon Cloud does not allow to upload your own apps. Server-side customization is currently only possible with a self-hosted Photon Server.

Pelekas

Final update: problem resolved.
Upgrading the network adapter drivers on the client pc did the trick.

For some reason I was confinced this was a problem on the router and I forgot about my network cards.

One thing I still don't understand: when the problem was on my client pc, how come putting my server pc on cable fixes this problem as well?
Some mysteries will have to remain to keep things interesting :-)

A big thank you all for your help!

[Deleted User]

Glad to hear that you found the solution!