Causes and how to handle photon warnings

Hi guys,

We have implemented photon V2 for our game and are working with our own game API above the photon layer.

We are lately getting allot of "QueueOutgoingAcksWarning", as well as some "QueueIncomingReliableWarning" and possibly "QueueOutgoingReliableWarning".

My question is what could be the cause of this and how should we address this. How severe are these warnings?
Note, I currently disconnect clients when this happens, and I think this is wrong but I urgently need to understand how to address this.

Also, I found it hard to find information about these and recommend adding some section that will advise on such matters as it is extremely important.

Cheers!

Comments

  • I am also wondering specifically:

    Lets say I keep getting these warnings for 5 seconds in a row.

    What happened to the messages sent and received by the client during this time?

    Do I know for a fact they will all be received on client or server once the warnings stopped? Were some possibly lost? All lost?
  • If you put enough stuff into them, then the queues will grow until the OS stops allowing your app to claim more memory, for example because its running out of it. This would be dimensions above the values, when the warnings trigger, so one can say "yes, all messages will still get queued and processed".
    The problem caused by big queuesizes, because of which we have built in these warnings, is a different one:
    Lets say, you are calling sentOutgoingCommands() once every 50ms, but you have 200 commands waiting in your outgoing queues: Then it will take 50ms*200==10.000ms==10s for the last of these commands to go out, after it has been added, which will most likely be an inaccepable lag.

    When you get this warning, then this means, that your are calling sendOutgoingCommands() or dispatchIncomingCommands() or both of them not often enough in comparison to how many (and how big in case they are big enough to get fragmented into multiple commands) messages you are sending and receiving. Both of these methods will also get called by service() (the first one in a loop, the second one only once), so calling service() more often will also help.
  • I currently use only Service() in the client.

    Other then calling it more often, will calling sendOutgoingCommands() on outgoing warning, and calling dispatchIncomingCommands() on imcomming warnings handle the crisis at hand?

    Also, should the service() in the client and the server be processed the same amount of time in a second?
  • gtidhar wrote:
    I currently use only Service() in the client.

    Other then calling it more often, will calling sendOutgoingCommands() on outgoing warning, and calling dispatchIncomingCommands() on imcomming warnings handle the crisis at hand?
    You should not get these warnings, after going live, but fix their cuases while developing. When you get such a warning, then there are already 100 items waiting in the queue, so even if your are immediately caring about the issue then, the player will already experience a serious lag.
    You would better check the QueuedIncomingCommands and QueuesOutgoingCommands properties of the peer and react, when you recognize the queues getting filled, instead of waiting for the warnings to trigger.
    gtidhar wrote:
    Also, should the service() in the client and the server be processed the same amount of time in a second?
    Not necessary.
  • I see.

    Generally speaking, the queue seems good. It can reach 10, 14 queue at times, but no more then that.
    Lets say the client has a disconnection on network level between it and the server for 2 seconds.
    Lets says I got a queue warning.

    But once I get back, I presume this should be fixed if everything else is working fine. Correct?

    (NOTE: our game is not an FPS or strategy. It's a training/learning game, and while the multiplier is crucial for the game to run, we can live with unexpected delays every once in a while. Even if it a 10 seconds one (if that does not happen too often). The crucial point for us is not losing the connection.
  • I still don't understand when will these messages be fired.

    Our queue is usually 0. In and out. When moving the avatar, it tends to get to 3 outgoing.

    Usually, on disconnections between the client and server, I get timeout messages or disconnection messages fired.

    What causes the warnings to show?
  • > We are lately getting allot of "QueueOutgoingAcksWarning", as well as some "QueueIncomingReliableWarning" and possibly "QueueOutgoingReliableWarning".

    QueueOutgoingAcksWarning: Happens when you client has 100 ACKs to send in it's queue. This happens when you lack SendOutgoingCommands calls, compared to the amount of data it receives and has to ACK.

    QueueIncomingReliableWarning: Happens when you get a lot of reliable data and don't dispatch it quickly enough. So this also piles up in-library and you might want to call DispatchIncomingCommands more often.


    QueueOutgoingReliableWarning: Happens when you created many reliable operations/commands to send on the client but didn't call SendOutgoingCommands() as much. This can also be caused by less commands with more content. Data > 1100 (approx) bytes is fragmented into multiple reliable fragment commands. Each counts vs the QueueOutgoingReliableWarning.


    Neither warning is a error. It's a warning.
    But they have side effects, if these happen constantly. There is lag happening and if ACKs stay in queue instead of traveling to the server, then a disconnect is imminent.
    You need to control how much you send and how often you call DispatchIncomingCommands() and SendOutgoingCommands(). And: You have to make sure that you DO call these. If you just "intend" to call them, a bug might prevent these calls still, so make absolutely sure you call these as long as you want the connection to stay intact.
  • Thanks Tobias.

    As it turns out, our server which does a periodical timer synch sending (for a game timer) - at some case on a specific server, sent 100+ messages to each client at one point instead of just 1 sync message.
    Hence the queue warning.
  • What sort of time syncing do you need?
    We have a low level timestamp, which just syncs 4 bytes worth of milliseconds. This is not a date but built in and should be the most accurate you could get.