Message limits per room enforced?

I'm thinking of porting my game to photon since the recent announcement of UNET's deprecation, but I've _always_ hesitated to adopt Photon because of the message limits.

Theoretically with a 500msg/sec/room limit the maximum players in a fast paced game room would be 7, with 10 msgs/sec, right? And That allows no room for any additional messages, RPC's, etc.

I guess my question is how exactly is this limit enforced?

My game is a VR game, and have spoken with the developers of Onward many times on their discord, and their game is built on Photon Cloud, and they support 5v5. That alone, if they were using a send rate of 10, would bust the 500 limit. On top of that I know they synchronize every gun shot with an RPC, and most guns fire 10x/sec, so clients could potentially be sending 20 messages per second.

Are they paying for some package with no message limits or something, and I just can't find it on the website? Do they get billed overages? Does it just not matter?

Comments

  • JohnTube
    JohnTube ✭✭✭✭✭
    edited September 2018
    Hi @legend411,

    Thank you for choosing Photon!

    The 500msg/sec/room is a soft limit highly recommended to have a fair use of our public cloud.

    This is a complex topic. First you need to know that any calculation done is just a theoretical estimation and may not reflect reality. We recommend building a Proof-of-Concept and use it to gather real data.

    That being said here is how to estimate messages/room/second generated by ALL players inside a room:

    Let's assume the following:

    - a room has N players.
    - a player sends F messages per second (message send rate in Hz)
    - every player multicasts messages to everyone else in the room (N - 1)

    Then we say that on average, the number of messages/second/room in your game as follows:

    R = N * [(N - 1) + 1] * F
    R = N * N * F
    R = N² * F

    if F = 10 (send rate) and N = 10 (5 + 5) ==> R = 100 * 10 = 1000

    There are ways to reduce that rate: e.g. Network/Traffic Culling (or Interest Management) using Interest Groups.
  • JohnTube said:


    if F = 10 (send rate) and N = 10 (5 + 5) ==> R = 100 * 10 = 1000

    Right, that was my point... Onward has to at least be sending 1000msg/sec, presumably more for firing RPC's and other events. I'd like to make a game similar to Onward, it seems they just get a pass for more than doubling the message limit, would I too?

  • JohnTube
    JohnTube ✭✭✭✭✭
    Hi @legend411,

    Onward is indeed a Photon powered VR game.
    They do not seem to go above the limits, so I would ask them how they manage to do that.

    I edited my previous post, my colleague @Kaiserludi spotted a misinformation:

    ACKs, connection handling (establishment, keep alive, etc.) commands and resends do not count as messages.
    messages == op requests + op responses + events
  • JohnTube said:

    Hi @legend411,

    Onward is indeed a Photon powered VR game.
    They do not seem to go above the limits, so I would ask them how they manage to do that.

    I edited my previous post, my colleague @Kaiserludi spotted a misinformation:

    ACKs, connection handling (establishment, keep alive, etc.) commands and resends do not count as messages.
    messages == op requests + op responses + events
    I have, they said they don't do anything in particular to optimize for msg/sec/room, so I don't really understand.

  • OneManArmy
    OneManArmy ✭✭✭
    edited September 2018
    * with 10 players you must lower sendRateOnSerialize to 5-6/s.
    * or if you have large maps, you can use Network Culling.

    In case you are ready to pay more, i think you can contact support and maybe you will come up with custom plan.
  • legend411
    legend411 ✭✭
    edited September 2018

    * with 10 players you must lower sendRateOnSerialize to 5-6/s.
    * or if you have large maps, you can use Network Culling.

    In case you are ready to pay more, i think you can contact support and maybe you will come up with custom plan.

    I just noticed on the plan page it tells you you get 3gb/CCU and you get charged for overages... I wonder if thats it how they handle it?

    Ultimately I think the message limit is dumb anyway, if I'm using 3000 msg/sec/room but each message is one byte, and some other game is using 500 msg/sec/room but 320 bytes per message, that game obviously costing photon much more than me.

    I compress the heck out of everything I sync, using UNET I send 20x/sec/player in an 8 player game, but I average about 600-800kb/sec per player (a bit more for the master client, and this includes all the UNET overhead which is significant compared to photon). Its peanuts, really.

  • As i understand higher message limit = requires more servers to process = you will pay more anyway.
    Question is only how it scales (message limit/price)?
    With 1.5K message limit, we could have 16 players (without any interest management).

    By the way any plans to increase msg limit?
  • JohnTube
    JohnTube ✭✭✭✭✭
    Hi @legend411,

    I wonder if thats it how they handle it?
    Yes you pay for bandwidth exceeding 3GB/CCU/month.
    Here is how to estimate that traffic.

    Hi @OneManArmy,

    Question is only how it scales (message limit/price)?
    I believe scaling is a factor of CCU.

    By the way any plans to increase msg limit?
    Not that I know of.
  • Kaiserludi
    Kaiserludi admin
    edited September 2018
    @JohnTube:
    If I understand @legend411 correctly, then he is asking if we would allow for a higher message limit if one pays for the additional traffic.
    @legend411:
    This is not how it works. The traffic overuse fees apply independently from the msg-limit. If you stay within the msg-limit, but still cause more traffic than what is included (because your average message size is so high), then you still need to pay traffic overuse fees. Don't worry: If someone is using significantly more traffic, then what is included in the pricing, then we usually inform them about this fact after a couple of days, so that they might take countermeasures to avoid a bill that is above their heads - we don't simply wait until the month is up and then present them a huge bill that they were not expecting.
    600-800kb/s per player is very, very far away from peanuts. 800kb per second results in 44MB per minute, 2.7GB per hour, 66GB per day and 2TB per month and this is for just one player. So each CCU would cause you 2TB of traffic (or 1.5TB for 600kb/s) per month, which is about 1000(!) times as much traffic as the average Photon-powered game causes.


    Ultimately I think the message limit is dumb anyway, if I'm using 3000 msg/sec/room but each message is one byte, and some other game is using 500 msg/sec/room but 320 bytes per message, that game obviously costing photon much more than me.

    There are two flows in your assumptions.
    1)
    There is no such thing as a 1 byte message. Your payload may be 1 byte, but there is protocol overhead for UDP, enet and Photon. 3000 messages with a payload of 1 byte each will cause a lot more traffic than 300 messages with a payload of 100 bytes each.
    2)
    It's not just about the traffic. Even considering the overall size of the messages including protocol overhead and not just the pure payload many small messages will cause the servers more work than fewer, but bigger, messages, even when they cause the exact same amount of traffic. More work for the server per client means that less clients can be served by one server and that we hence need more servers to serve all clients, which results in higher costs.
    More messages also means more work for the clients and with too many messages especially lower-end mobile clients might simply not be able to keep up anymore. With your 600-800kb/s you are already making it impossible for a lot of clients to be able to keep up independently of the message-size just because their bandwidth is lower than that, by the way. 800kb/s is 6.4mbit/s. There are still a lot of clients out there that have less than 1mbit of available bandwidth, even on landlines, let alone mobile networks.
  • legend411
    legend411 ✭✭
    edited September 2018


    There are two flows in your assumptions.
    1)
    There is no such thing as a 1 byte message. Your payload may be 1 byte, but there is protocol overhead for UDP, enet and Photon. 3000 messages with a payload of 1 byte each will cause a lot more traffic than 300 messages with a payload of 100 bytes each.
    2)
    It's not just about the traffic. Even considering the overall size of the messages including protocol overhead and not just the pure payload many small messages will cause the servers more work than fewer, but bigger, messages, even when they cause the exact same amount of traffic. More work for the server per client means that less clients can be served by one server and that we hence need more servers to serve all clients, which results in higher costs.
    More messages also means more work for the clients and with too many messages especially lower-end mobile clients might simply not be able to keep up anymore. With your 600-800kb/s you are already making it impossible for a lot of clients to be able to keep up independently of the message-size just because their bandwidth is lower than that, by the way. 800kb/s is 6.4mbit/s. There are still a lot of clients out there that have less than 1mbit of available bandwidth, even on landlines, let alone mobile networks.

    Gah, sorry, I meant 600-800 bytes/sec. Typo lol.

    I understand packet headers and obviously my scenario was an over simplification, I don't want to send 3000 messages per second, but I think 1000, at 600-800 bytes/sec per client would be reasonable.

    It's obvious you let some people do it, there's no way a game like onward would work with a 5 tickrate. Even on Mike Hergaarden's blog he blatantly admits he uses +-1000 messages/sec on photon cloud AFTER implementing network culling (https://www.m2h.nl/network-traffic-culling/ "from 2000 messages per second to +/- 1000") in Verdun.

    I'm just curious what the actual process is, it needs to be clearer before I spend weeks porting my game from UNET. Do people get flagged if they hit 501? If the game occasionally spikes to 800-1000 messages/sec and then goes back down to 500, is that okay?

    The UNET relay had a limit of 4kb/sec per client, that was so much easier to keep track of and stay under, and be able to at least make a 4v4 game with a 10 sendrate.

  • Kaiserludi
    Kaiserludi admin
    edited September 2018
    Hi @legend411.


    Gah, sorry, I meant 600-800 bytes/sec. Typo lol.

    Ah, that sounds much better. With that number you would actually stay within the included traffic.

    As @JohnTube has already mentioned in his initial response, the 500msg/s is a soft limit.
    This means that we don't have some automatic technique in place to enforce the limit and to let the server immediately send an error response to the clients whenever a room reaches 501 msg/s.
    Rather we measure usage with monitoring tools and if an app is consistently and significantly above the limits and has relevant CCU, then we will manually contact the developer of that app and talk to them about this issue and work out a solution on a case by case basis.

    For example such a solution could be that the developer subscribes to twice the amount of CCU that he actually generates and in exchange is allowed to generate twice as many msg/s per CCU than he would otherwise be allowed to.
  • Hi @legend411.


    Gah, sorry, I meant 600-800 bytes/sec. Typo lol.

    Ah, that sounds much better. With that number you would actually stay within the included traffic.

    As @JohnTube has already mentioned in his initial response, the 500msg/s is a soft limit.
    This means that we don't have some automatic technique in place to enforce the limit and to let the server immediately send an error response to the clients whenever a room reaches 501 msg/s.
    Rather we measure usage with monitoring tools and if an app is consistently and significantly above the limits and has relevant CCU, then we will manually contact the developer of that app and talk to them about this issue and work out a solution on a case by case basis.

    For example such a solution could be that the developer subscribes to twice the amount of CCU that he actually generates and in exchange is allowed to generate twice as many msg/s per CCU than he would otherwise be allowed to.
    Ahh that makes sense! And seems totally reasonable. Guess I'm a photon guy now! Thanks for your responses, Kaiser.

  • Besides the more detailed counting above, a few general notes and summary on msg/s:

    - we do not have a hard cap on 500 msg/s per room, so you can go over a bit
    - when going over the limit
    (i) pay extra close attention to your used traffic (and check traffic overage cost)
    (ii) make sure your clients can cope with the amount of messages (especially mobile clients might get issues here)

    As long, as your higher msg/s rate does not have a negative effect on other apps or consumes too many resources (server CPU, mem., I/O), we are ok with it. If we think your app has a negative effect on others because of increased msg/s, we will ask you to adjust it.