Cloud: Disconnection during level loading

KevDK

Hey Exit Games,

We're in the process of converting our Unity-based networking system into Photon. We've made good progress so far, but we've run into a problem related to level loading. Here's the scenario:

Photon: Photon Unity Networking, downloaded from the Unity Asset Store. Connected to the Photon Cloud.
4 players join a room, this makes them all load a nearly empty scene (let's call it Scene A). So far so good. This acts as our room lobby (where we wait for all players to connect before properly starting the game - no rush here). When the MasterClient has confirmed that all 4 players are connected and ready, the MasterClient signals out to all clients (including itself) to switch to Scene B, using PhotonNetwork.LoadLevel(SceneBName). Here's our problem: SceneB is rather big, and if any of our clients uses more than 10-15 seconds to load SceneB, Photon forcibly disconnects them (with the StatusCode DisconnectByServer). This should not happen.

According to the webpage, there are the following suggestions for this scenario:

The Photon Unity Networking package implements the Service calls for you: PhotonHandler.Update() calls Service in intervals.

However, Unity won't call Update() while it's loading scenes (or maybe even any assets).

To keep the connection while loading scenes, you should set PhotonNetwork.IsMessageQueueRunning = false.

Pausing the message queue has two effects:

A background Thread will be used to call SendOutgoingCommands while Update is not called. This keeps the connection alive, sending acknowledgements only but no events or operations (RPCs or sync updates). Incoming data is not executed by this Thread.
All incoming updates are queued. Neither RPCs are called, nor are observed objects updated. While you change the level, this avoids calling RPCs in the previous one.

We've tried a lot of different configurations, but it just doesn't seem like the background thread is actually working (it's hard to see where it's actually set up in the code?). We tried using the default PhotonNetwork.LoadLevel (which we can see sets PhotonNetwork.IsMessageQueueRunning = false, and is later set to true in NewSceneLoaded). The problem persists. We tested manually setting it to false before and true after loading, but the client is disconnected before loading finishes. Also, if we manually set PhotonNetwork.IsMessageQueueRunning = false (and do nothing else while in SceneA, just for testing purposes), the client is being kicked out after 10-15 seconds, regardless of what values we assign to both PhotonNetwork.networkingPeer.DisconnectTimeout and PhotonNetwork.networkingPeer.SentCountAllowance. This might be because the Cloud Master Server/Game Server has its own disconnection parameters?

To summarize, it looks like the background thread isn't running (or something similar), which makes our clients disconnect if they're loading for too long. Is there anything, we're missing here? If we had used the plain Unity SDK, I could see that we'd need to set up a seperate thread, but the documentation on the webpage suggests that this should be in place already.

Tobias

Which version of PUN are you using? Which Unity version and platform do you run this on?

I can take a look if the Thread is running but it should.

You can check the client's performance by enabling the TrafficStats:
PhotonNetwork.NetworkStatisticsEnabled = true;
To check gaps for sending, use: PhotonNetwork.NetworkStatisticsToString() or show this somewhere: PhotonNetwork.networkingPeer.TrafficStatsGameLevel.LongestDeltaBetweenSending

Maybe it's something else entirely.

KevDK

We're using Photon Version 1.22.2, dated at 13th of August. We're using Unity version 4.2.0f4 (so it should be up to date).

I've hosted a room with a single client (the Master Client). It stays in SceneA the whole time. At a point of time, we set PhotonNetwork.isMessageQueueRunning = false; The image shows the results:



The first section is with PhotonNetwork.isMessageQueueRunning = true. Everything looks normal - Call of Sends and Dispatches are going up normally. Ms since last receive caps at 990 ms.

The second section is the moment we set PhotonNetwork.isMessageQueueRunning = false. Calls of Send and Dispatch immediately stop rising.

The third section is after 25 seconds of waiting. The Ms since last receive first caps at 2000 ms, the next caps at 4000 ms, and after that, it keeps going up unhindered (Disconnected at this point).

BoH_Havoc

Had the same problem 2 days ago.

I'm using A* Pathfinding by Aaron Granberg and am generating a NavMesh after Level Load (can take up to 20 Seconds). During the NavMesh generation, the game more or less freezes. Setting isMessageQueueRunning to false didn't stop the game from disconnecting.
After starting my own thread with Photon Update logic (which runs during isMessageQueueRunning == false), there were no more disconnects and it started working as it should.

So either we are both doing something wrong, or there is something wrong with the current version of Photon/Unity.

I'm using the latest Unity 4.2.x and the latest PUN from the Asset Store.

Tobias

If you set IsMessageQueueRunning to false before you start loading, an extra Thread should run the SendAcksOnly method of the photon peer. This should keep the connection, even while Unity is not even calling Update().
You need to set IsMessageQueueRunning to true again, after loading.
PUN's LoadLevel should do this by itself.

I am in Vancouver and limited in time to test but if this is a problem for you, I am pretty sure we can fix this until end of next week.

Does anyone have a minimal repro project? If so, please send it to developer@exitgames.com (put a link to this thread in the mail pls) and I'll take a look asap.

KevDK

If you set IsMessageQueueRunning to false before you start loading, an extra Thread should run the SendAcksOnly method of the photon peer. This should keep the connection, even while Unity is not even calling Update().
You need to set IsMessageQueueRunning to true again, after loading.
PUN's LoadLevel should do this by itself.

Is the Thread supposed to start the moment you set IsMessageQueueRunning = false, or is it supposed to start the moment you call PhotonNetwork.LoadLevel? Making a repro project that shows off the first is rather easy. The second is a bit harder to replicate, since it depends on the computer specs and the scene we're trying to load.

KevDK

In the lack of response, I've created a project that assumes that the first point above is the interesting one (setting IsMessageQueueRunning to false shouldn't disconnect you) - the Unity project can be downloaded here.

Please keep us posted on your findings, and if you need anything else

Tobias

Cool you created a test case. Thanks for that!
I think I found something. The background thread is started but it checks if it should do something. However, this always has a value of false.

In PhotonHandler.StartFallbackSendAckThread, add the second line:
[code2=csharp]public static void StartFallbackSendAckThread()
{
SupportClass.CallInBackground(FallbackSendAckThread);
sendThreadShouldRun = true;
}[/code2]

I will fix this part in the next PUN update, so you don't have to worry about keeping this update in.
Let me know how that works for you, ok?

KevDK

I can confirm that this works - we had an old Mac running where it took 2+ minutes to load the game,and it didn't disconnect. We can also see that the thread keeps the connection running in the room when we just turn off isMessageQueueRunning and wait. Case closed

Tobias

Thanks for the confirmation!

Claytonious

Has this fix been included the current "plain" SDK that is here http://doc.exitgames.com/photon-cloud/SDKunity3d/ ?

Tobias

Cleytonious: The plain SDK does not have this problem. There is no similar background Thread in it and assumes you handle it.
Let me know if you're stuck somewhere.