Idle Android players

robertsze

We still have a major issue with Idle Android Players. (as reported on the unity3d forum already). We did now more tests with around 700 CCUs and 100 rooms running. Whenever Android puts the app in background we can't disconnect the players automatically in OnApplicationPause() because this would also disconnect players when for example a "low battery" message pops up, the user watches a video or purchases an in-app. When the player "never" returns the player never gets removed from the room.

What we do now is to let every other client detect idle players (players not responding to a heartbeat message for 60 seconds) and simulate a disconnect of this client on all other clients. As said, this works very well, but there is one major problem. The not responding client doesn't get disconnected from the photon server, meaning the player still occupies a slot in the room. In the worst case, a room is than filled with only idle players and no one can join anymore.

We can think of two possible solutions for this issue, but both require help from you photon guys.

1. Kick players. The current CloseConnection() doesn't work as this method relies on the idle player to actually kick himself. (CloseConnection just sends a message to the client which then kicks itself. Works fine as long as this player is still responding). If there would be a command which we can send to the server which actually kicks the player and informs all other players about the disconnect it would be fine.

2. Heartbeat control by photon itself. For example, every client has to send a specific heartbeat message to photon server. If the server doesn't receive such a message in a specified amount of time, than photon kicks the player like in the first case.

Hope we can get a solution for this issue quickly... thanks guys!

vadim

Hi,

When running in background, android client sends pings to the server to keep connection alive.
You can try to disable this. Client should be disconnected by server after several seconds of inactivity.
Comment out line with SupportClass.CallInBackground() call in StartFallbackSendAckThread() method in \Assets\Photon Unity Networking\Plugins\PhotonNetwork\PhotonHandler.cs

robertsze

vadim said:

Hi,

When running in background, android client sends pings to the server to keep connection alive.
You can try to disable this. Client should be disconnected by server after several seconds of inactivity.
Comment out line with SupportClass.CallInBackground() call in StartFallbackSendAckThread() method in \Assets\Photon Unity Networking\Plugins\PhotonNetwork\PhotonHandler.cs

Ah, there is a background thread pinging the server. (wasn't sure about this, thought the server would just remove the player when the TCP/UDP connection was closed).

I will give this immediately a test, but have two questions if you don't mind.

1. If I disable this background thread, do I have to manually send acks regularly to the server? (if we don't send other commands to the server regularly).

2. Is it possible to configure the time until the server kicks the player? (This would be required so that users can successfully complete in-apps, watch videos, cancel incoming calls, etc...), but will be kicked if he has the app in the background for a longer time.

Thanks!

robertsze

Ok, so I tried what you suggested and this is mostly working.

I have to manually send the Acks so that the server doesn't drop the connection. The server kicks the player when we are in the background for 10 - 15 seconds. If there now would be just the possibility to adjust this timeout time everything would be fine.

On a side note, this tests resulted in an unhanded exception inside photon.

NullReferenceException: Object reference not set to an instance of an object
NetworkingPeer.OnSerializeRead (ExitGames.Client.Photon.Hashtable data, .PhotonPlayer sender, Int32 networkTime, Int16 correctPrefix) (at Assets/Photon Unity Networking/Plugins/PhotonNetwork/NetworkingPeer.cs:3519)
NetworkingPeer.OnEvent (ExitGames.Client.Photon.EventData photonEvent) (at Assets/Photon Unity Networking/Plugins/PhotonNetwork/NetworkingPeer.cs:1936)
ExitGames.Client.Photon.PeerBase.DeserializeMessageAndCallback (System.Byte[] inBuff)
ExitGames.Client.Photon.EnetPeer.DispatchIncomingCommands ()
ExitGames.Client.Photon.PhotonPeer.DispatchIncomingCommands ()
PhotonHandler.Update () (at Assets/Photon Unity Networking/Plugins/PhotonNetwork/PhotonHandler.cs:92)

It appears that the crash happens at:

...
if (sender.ID != view.ownerId)
...

inside NetworkingPeer.OnSerializeRead().

Whats interesting is this function:

PhotonHandler::Update()
{
....
       bool doDispatch = true;
        while (PhotonNetwork.isMessageQueueRunning && doDispatch)
        {
            Profiler.BeginSample("DispatchIncomingCommands");
            doDispatch = PhotonNetwork.networkingPeer.DispatchIncomingCommands();
            Profiler.EndSample();
        }
...}

The second last callback we get from this loop is:
NetworkingPeer:OnStatusChanged(StatusCode) which tells us that the connection run into timeout. (exactly what we want).

I guess at this time all photon views and players are destroyed already.

Then DispatchIncomingCommands process another message, the one which leads to the exception above. If I understand correctly nothing should handled anymore in this case as the connection is done already. Hope this helps!

vadim

Thanks for detailed report. We will look at the issue as soon as possible.
Sorry, disconnect timeout is not adjustable currently.
I hope current workaround will work for you until we handle this properly.

robertsze

Unfortunately this is a major issue for us at the moment, if I can't get this fixed we would have to take Stickman Battlefields offline. Anyway, I have one more idea, please let me know what you think about it and if you can think of any side effects:

My idea would be to add a timeout handling directly in the Android background thread. At OnApplicationPause() i reset the pause counter, increment it by 100msec in the thread (I can't use a counter form an update routine because they are suspended from unity, nor can I use realtimeSinceStartup as this must be called from the main thread). If the counter reaches a specified threshold I stop sending Acks. If I want a timeout for for example 60 second I would set this threshold to 50. (50 + 10 seconds for the server to disconnect).

Something like this:

    public static bool FallbackSendAckThread()
    {
	    if (sendThreadShouldRun && PhotonNetwork.networkingPeer != null)
	    {
		    if (pauseActive)
		    {
		    	pauseTime += 100f; // ugly, but assume 100msec
		    	
			    if (pauseTime > (pauseKickTime-10)*1000) // ugly, but assume 10 secs for photon disconnect
	        	{
				    Debug.Log("Stop sending ack to server, idle for too long");
				    return sendThreadShouldRun;
	        	}
		    }
	        PhotonNetwork.networkingPeer.SendAcksOnly();
        }

        return sendThreadShouldRun;
    }

To workaround the NullReference issue I use:

  public void OnEvent(EventData photonEvent)
  {
  	    // modified
	    if (PhotonNetwork.connected == false)
		    return;
	    ......
  }

What do you think about the two workarounds? Any possible side effects? We could "live" with this hack for now

Kaiserludi

Hi robertsze.

Couldn't you just disconnect the players automatically in OnApplicationPause() and reconnect, when the application comes back to foreground?
When it is important to your game that it disconnects a client, when the app goes to background on that client, then it should always do so, including scenarios when for example a "low battery" message pops up, the user watches a video or purchases an in-app as this is completely irrelevant for the other players. They can't see on their devices, if that user surfs youtube or if he put his phone aside nor do they care. All they know and care about is that he isn't there anymore. Also watching a video or doing in app purchases could also last for hours and a low battery message means, that the device of that user might soon run out of battery entirely, so for all those scenarios it makes sense to handle them exactly like other background scenarios: either ignore them completely and let the players idle or forcibly disconnect every client in OnApplicationPause() or disconnect everyone after a certain period of inactivity time.

When disconnecting in OnApplicationPause() you could even sen a message out to the other clients before doing the disconnect to let them know that that client turns into background. Those other clients could then simply not present the disconnect to the user for a certain amount of time and do not so at all, if that client reconnects in that timespan. Alternatively you could basically do a kind of reverse version of your current heartbeat logic: Whenever you see a disconnect, just give that client 60 seconds for a chance to reconnect and send a message to the other clients, before presenting that disconnect to the player.

Although it is not currently possible with Photon Public Cloud to set the server side timeout, this is possible with Photon Server and with photon Cloud Enterprise, so switching to one of those may also be an option for you.

robertsze

Thanks for the feedback.

We considered this too. Silently disconnecting and reconnecting when the user comes back up to 60 seconds later. But this makes things much, much more complicated. In the worst case the last room slot is even occupied by another user then. Problem is, it is completely understandable that customers complain about disconnects when there just appears a "Running soon out of battery" message, or a new mail arrives, or they decide to make an in-app purchase. I don't talk about a user who decides to watch a video in youtube. Disconnecting immediately in this case would be fine. I talk about activities running inside the app, like a incentivized video activity, google in app purchase dialog, even a simple dialog activitiy. This is where the player gets disconnected if we use OnApplicationPause(), and I think it is understandable that this upsets customers. If we don't disconnect them in OnApplicationPause(), the player will idle forever.

I think the solution to this entire issue would be rather simple. Disconnect automatically if in background for more than X seconds. So the user always has a chance to dismiss message boxes, can even watch aa 30 second incentivized video (not talking about youtube in this case), and can even purchase a in-app. The user can even just push the power button and put the phone away, other players will see him disconnecting automatically after X seconds.

The problem is not photon itself, but android. The way android handles activities. But I fear photon is the only one being able to handle this. Compare iOS for instance, there the system behaves correct:
Whenever you show a incentivized video, a in-app purchase dialog, a rate-this-app-dialog or the battery message comes app, iOS will NOT CALL OnApplicationPause(). Everything just works fine. Only as soon as you press the home or power button on the iPhone/iPad, OnApplicationPause() is called. This is what the user and developer expects eventually. But on Android OnApplicationPause() is called when the user actively pauses the app by pressing home or power button, but also when a simple dialog box appears.

Hope you get what I mean...

Kaiserludi

Thanks for the additional explanations.

We will discuss this topic internally after the current Unite is over and all our team members that are currently at the Unite are back in the office.

robertsze

Just for you information:

We handle this issue now locally right inside PhotonHandler with something like this:

public static bool FallbackSendAckThread()
{
	    if (sendThreadShouldRun && PhotonNetwork.networkingPeer != null)
	    {
	    	// djinnworks start
		    if (PhotonNetwork.pauseActive)
		    {
		    	PhotonNetwork.pauseTime += PhotonNetwork.pauseHandlerLoopTime;
		    	
		    	float maxTime = (PhotonNetwork.pausePhotonDisconnectTime - PhotonNetwork.pausePhotonDisconnectTimeServer) * 1000f;
		    	if (maxTime < 5f * 1000f)
			    	maxTime = 5f * 1000f;
			    
			    if (PhotonNetwork.pauseTime > maxTime)
	        	{
				    Debug.Log("Stop sending ack to server, idle for too long");
				    return sendThreadShouldRun;
	        	}
		    }
		    // djinnworks end
	        PhotonNetwork.networkingPeer.SendAcksOnly();
        }

        return sendThreadShouldRun;
}

We set pauseActive and pauseTime right inside OnApplicationPause right inside PhotonHandler. This works perfectly. Tested with more about 1000CCUs and thousand of played games, not "idling" players anymore.

What we do in addition to this is to switch the masterClient even earlier (after 10 seconds) of timeout with a custom heartbeat. This way logic which is just handled by the master client will not be affected for too long when a player switches his app into the background.

Combing this both methods we got stable and perfect multiplayer gaming now.

kamend

Hey @robertsze, could you share the other part of the solution, I am curious about PhotonNetwork.pauseHandlerLoopTime, do you update this in another thread or?

Thanks!

robertsze

There you go:

public static class PhotonNetwork
{
	// djinnworks start
	public static bool   pauseActive;
	public static float  pauseTime;
	public static float  pauseHandlerLoopTime            = 100f;
	public static float  pausePhotonDisconnectTimeServer = 10f;
	public static float  pausePhotonDisconnectTime       = 60f;
	// djinnworks end
        ......
}


internal class PhotonHandler : Photon.MonoBehaviour, IPhotonPeerListener
{
        ......
        ......
        // djinnworks start
	public void OnApplicationPause(bool pause)
	{
		if (pause == true)
		{
			PhotonNetwork.pauseActive  = true;
			PhotonNetwork.pauseTime    = 0f;
		}
		else
		{
			PhotonNetwork.pauseActive  = false;
			PhotonNetwork.pauseTime    = 0f;
		}
	}
	// djinnworks end	
}

pauseHandlerLoopTime is set to 100msec (which is the rate at which the android background calls the FallbackSendAckThread() method

robertsze

pauseHandlerLoopTime (100msec) and pausePhotonDisconnectTimeServer (10sec) are assumed to be fixed at the values currently used by Photon and they hopefully never change

kamend

Thanks!

Tobias

Thanks everyone for the discussion and actually posting a solution.
Based on your input, I added something similar to PUN v1.62, which should be in the Asset Store shortly.
Hope it works as you expect.

robertsze

Awesome!! Will try it as soon as it is live!