Getting an infinite loop within DispatchIncomingCommands causing the game to hang

I'm looking into an issue where occasionally in our Unity game we're experiencing some kind of hang.

I narrowed it down to a single call of DispatchIncomingCommands getting into some kind of infinite loop.

With high levels of logging, it ends up spitting out something like this for what seems like forever before I have to kill the process:

command was received before! Old/New: NC(0|7 r/u: 761/7017 st/r#/rt:0/0/0)/NC(0|7 r/u: 761/7017 st/r#/rt:0/0/0)

command was received before! Old/New: NC(0|7 r/u: 761/7018 st/r#/rt:0/0/0)/NC(0|7 r/u: 761/7018 st/r#/rt:0/0/0)

command was received before! Old/New: NC(0|7 r/u: 761/7019 st/r#/rt:0/0/0)/NC(0|7 r/u: 761/7019 st/r#/rt:0/0/0)

command was received before! Old/New: NC(0|7 r/u: 761/7017 st/r#/rt:0/0/0)/NC(0|7 r/u: 761/7017 st/r#/rt:0/0/0)

command was received before! Old/New: NC(0|7 r/u: 761/7018 st/r#/rt:0/0/0)/NC(0|7 r/u: 761/7018 st/r#/rt:0/0/0)

command was received before! Old/New: NC(0|7 r/u: 761/7019 st/r#/rt:0/0/0)/NC(0|7 r/u: 761/7019 st/r#/rt:0/0/0)

command was received before! Old/New: NC(0|7 r/u: 761/7017 st/r#/rt:0/0/0)/NC(0|7 r/u: 761/7017 st/r#/rt:0/0/0)

The call stack looks something like this:

(Unity Debug.Log stuff)
Photon.MmoDemo.Client.Game:DebugReturn(DebugLevel, String) (at Assets\Plugins\Game.cs:573)
ExitGames.Client.Photon.EnetPeer:QueueIncomingCommand(NCommand)
ExitGames.Client.Photon.EnetPeer:ExecuteCommand(NCommand)
ExitGames.Client.Photon.<>c__DisplayClassc:b__a()
ExitGames.Client.Photon.EnetPeer:DispatchIncomingCommands()
ExitGames.Client.Photon.PhotonPeer:DispatchIncomingCommands()
(Game code -- effectively a Monobehaviour Update())

At first I thought it was a Service() call getting into an infinite loop, so I broke it up and did a naive time slice to prevent an infinite number of calls to DispatchIncomingCommands, but that didn't fix the issue.

Any thoughts?

Comments

  • I've tried to use PhotonPeer.CommandInfoCurrentDispatch to dig more into what it's currently processing when it starts spitting out that message, but that string seems to be empty. There seems to be a comment in the documentation that it's always empty in a release build of the lib-- is there a debug version of the Photon3Unity3D dll available somewhere?
  • JFlanagan
    JFlanagan
    edited January 2016
    Well a bit of an update.

    Since everything seemed to be black boxed for me I went ahead and decompiled the Photon3Unity3D DLL so I could debug some stuff.

    What looks like is happening is not that there's an infinite loop, per se, but the ActionQueue in EnetPeer ends up having (in one case) 500,000+ commands in it. There's just some burst of data the socket client seems to receive, which calls EnqueueActionForDispatch a ton of times.

    The odd thing is that the burst of data the client is receiving gets up to something like the mbit/second range, but looking at the server it doesn't seem to be transmitting anywhere near that much information (if Resource Monitor can be believed).

    Is there any way to look at the data coming in to see what exactly it's retrieving -- be it event codes or whatever? Specifically I'm thinking if there's a way to look at the buffer in EnetPeer.ReceiveIncomingCommands.
  • More updates:

    So for whatever reason, when I get into this state it I'm getting QueueOutogoingAcksWarning. I was curious how many acks I was getting, so and it ended up being something like 20,000+. During this time the action queue is getting filled up with ping commands.

    After the acks are all queued up, then that's when the the action count itself starts incrementing. The numbers of events I'm seeing don't really jive with the number of events I believe the server is sending down. The client appears to be getting like 10,000+ move events (unreliable) in the span of a few seconds, but the server has caps on all that stuff that shouldn't put the numbers anywhere near that.

  • For the time being what I ended up doing is putting in a workaround that breaks the ActionQueue loop in DispatchIncomingCommands if it takes longer than X ms to run.