message time out question

KevinB
KevinB
edited January 2014 in Native
Hi,

here's a time out log of photon ping msg(I believe it's ping msg coz i wasn't sending anything)
2013-12-06 16:01:58,308053 INFO EnetPeer.cpp sendOutgoingCommands() line: 175 - going to resend pCommand: time: 111529 = 111436 sentCount: 1, original sentTime: 111313
2013-12-06 16:01:58,642201 INFO EnetPeer.cpp sendOutgoingCommands() line: 175 - going to resend pCommand: time: 111864 = 111775 sentCount: 2, original sentTime: 111313
2013-12-06 16:01:59,191534 INFO EnetPeer.cpp sendOutgoingCommands() line: 175 - going to resend pCommand: time: 112413 = 112356 sentCount: 3, original sentTime: 111313
2013-12-06 16:02:00,241524 INFO EnetPeer.cpp sendOutgoingCommands() line: 175 - going to resend pCommand: time: 113463 = 113397 sentCount: 4, original sentTime: 111313
2013-12-06 16:02:02,258454 INFO EnetPeer.cpp sendOutgoingCommands() line: 175 - going to resend pCommand: time: 115480 = 115431 sentCount: 5, original sentTime: 111313

my current timeout setting is 5 retry or 15sec, so the log shows that it timed out after 5 retry. The first retry happens ~123ms after the ping is sent. Our ping frequency is 100ms(10 times a second). 123ms seems short as the first time out or is it not? and I have seen the first retry time to be varied, something in the range of 300ms. How is the first retry time being determined?

also what's the priority of the actual thread that's sending out packets when compared to the main thread in iOS and Android?

Thanks!

Kevin

Comments

  • Hi Kevin.

    The first retry gets done after the rtt + 4 times the rttVariance has been passed, both determined at the moment, when the command originally has been send.
    Every further retry of the same command will then be sent after twice as much time as the previous one.
    So if you have an rtt of 100ms +/- 5ms, then the first 5 retries will happen after 120ms, 240ms, 480ms, 960ms and 1.920ms

    The priority of the actual thread that's sending out packets when compared to the main thread in iOS and Android is exactly the same as the one of the main thread itself, because packet sending gets actually done in the main thread (inside of sendOutgoingCommand(), which gets called by service(), but can also be called directly).
  • Thanks,

    have you guys seen any disconnect issue when using wifi? we have users from the field reporting frequent disconnection when using wifi. when we examine their round trip time and variance, they seem ok.

    Kevin
  • Hi again.

    We have experienced, that some wifi routers just seem to drop a lot of udp packets without any apparent reason, even if the connection quality looks great. For example in our office my colleague Tobias has done some testing with an android phone and two different wifi routers. The results were astonishing: for one router the phone displayed to only have a poor connection quality, but he barely had any packet loss at all, for the other router the phone showed that it would have a perfect connection quality, but surprisingly the packet loss rate has been that high, that it triggered a timeout disconnect every 4 hours. With 16 players in a room this means, every 15minutes one of them looses its connection.

    Sadly we can't really fix this: We would have to convince your players to exchange their wifi routers against different models that don't su*k.
    So that means that you have to experiment with workarounds and see what gives the best results for your game:
    - increase the max retry count - with the downside that the client of course also will need more retries before reporting a connection loss, when the connection is really down
    - switch to tcp - some wifis seem to perform better with tcp than with udp
    - increase the timeout settings on client and server side (not possible, if you use the public Photon Cloud)
  • Thanks,

    this is very helpful. surely it gives us some direction to look at. what router models would this disconnection issue occur? we would like to try a few and hopefully work around it.

    it sounds like tcp would be a good alternative? any concern if we go this route? a lot more packet overhead? also, can the server support both udp and tcp simultaneously? if not, then it's a bit tricky to do the transitioning of live clients from udp to tcp.

    our current retry count(5) and timeout settings(15s) seems to be reasonable. 15s is quite a lot for a real time game to be out of sync with the server, not sure if tweaking this value would help in our game play.

    Thanks,
    Kevin
  • Sorry, I can't recall the router router models, but I will ask Tobias to have a look at this thread.

    Downsides of using tcp instead of udp:
    - you can't choose per message anymore, if it should be sent unreliable or reliably, as with tcp every message always is reliable
    - you loose some low level control, as you can't explictly set message priorities by using different channels, which is possible when using udp in Photon

    Yes, the server can support udp and tcp clients simultaneously. They can join the same rooms and a client doesn't even know, which protocol is used by another client in the same room. In general it may be a good idea to have a game option where the player can choose the protocol. Make udp the default and offer tcp as a fallback option that the player can try, when he experiences issues with udp. That's also useful for other scenarios like for firewalls: Some people that would like to play your game may sit behind a firewall or router for which they don't have control over which ports are open. Sometimes these people are able to connect through tcp, but not trough udp and vice versa.

    Well, in my example from Monday 5 retries mean 2 seconds for the last retry, so you would still be far below 15 seconds in that case. You could set the retry count higher, even that high that it could never trigger before the the 15sec timeout. Your clients still wouldn't try longer than 15sec, but if their latency is good enough, then they would do more retries in that amount of time, which could maybe help in some cases, but of course they then would always need the full 15 seconds before timing out, while now a client with good latency may already timeout much earlier.
    For stable connections it may be the better choice to time out relatively quickly and don't wait for the full 15seconds as when there has barely been any package loss then with 5 lost retries for a single command there isn't much hope anyway that the connection has not been lost.
    For unstable connections on the other hand it may be better to give the clients a few more retries, although that means that timeouts are triggered later: When the packet loss rate of a clients connection is that high that it looses 3 retries in a row every few seconds, then a max of 5 retries may not be enough.
    You may even offer an advanced option in the games settings to let the player choose how often its client should try to resend, but I am afraid that may get too technical for a lot of players.
  • Hi Kaiserludi, do u get a chance to talk to Tobias to get a list of routers that drop udp packets? Thanks!

    K
  • Hi Kevin.

    Yes, I did, but he can't remember about the details.
  • We don't have a proper list. We only heard about issues with "Linksys WRT54G" by one of our customers.
    Others will break datagrams too but we don't know which models nor why they do in the first place.

    One thing that will improve things with our reliable UDP protocol is the "CRC check" feature we built in. It tries to detect broken datagrams before they lead to disconnects. Even though it means a few more re-sends, it should keep more connections. As broken packages are still loss, timeouts might happen still, of course.