Exceeding bandwidth limits by a lot.

PhotonUser
2021-06-18 16:58:22

I am working in a game where physics is crucial, players are active ragdolls, so each player has 15 rigidbodies to sync over the network, put 10 players on a server and it will add up exponentially.

Each rigidbody, if sent fully would be 3 Vector3, 1 Quaternion, total: 16 * 3 + 20 = 68 bytes.
So each player would send 1024 Bytes (68 Bytes * 15 rigidbodies) per message.
Put that 10 times a second and you have 10KB/s per player of send data.
In a 10 player match that would cost the server about 1MB/s since it needs to send 100KB/s (10 players * 10KB/s to other 10 players) because every player needs to know about eachother.

I managed to get that down to 3KB/s per player with some compression and other ideas, probably could get to 2.5KB/s if SerializationRate is 8 and deactivated some rigidbodies in cost of a little worse simulation.

Let's stick to 3KB/s example and 20CCU/60GB plan:
Server is now sending 30KB/s to each player, so 300KB/s of traffic for a 10 player match.

Then I figured out if I am running 10 player matches all day, that would cost me 25GB per day which is definitely gonna pass the plan's limit by hundreds of GB.

From my math 60GB/month equals 23KB/s.

I am interested in knowing how the Exit Games team estabilished the 3GB per CCU estimation, is that based on average play time per month?

Am I doing some wrong math here?

Comments

Kaiserludi
2021-06-18 18:33:00

Hi @PhotonUser.

The 3GB per CCU are based on the traffic consumption of an average Photon Cloud app: The value has been chosen so that most apps stay below it. If we would raise the amount of included traffic per CCU, then we would have to raise the price considerably. Including twice the amount of traffic would result in approximately twice the price, as traffic is the main cost driver. As most apps don't consume that much traffic, we don't include more than 3GB per CCU to be able to offer lower prices than what we otherwise could offer.

Developers, who want to consume more traffic and are OK with paying for their higher consumption, can simply subscribe to more CCU than they have, to raise their overall amount of included traffic, or they can choose to pay overuse fees for the amounts of traffic by which they surpass the amount included in their subscription.

As you will notice, if you do the math, you also surpass the recommend maximum amount of messages/s by a factor of 2.
This is not surprisingly as you are having 10 players per room and as when every player sends everything to everyone else in the same room, then the traffic and messages/s increase squared to the amount of players per room.
For this reason it is absolutely critical for games with more than 8 or so players, and when sending a lot of data per player, then even with more than maybe 4 or so players, per room, to NOT send every message to everyone else inside the same room, but to implement some form of interest management:
The further two players are apart from each other, the lower the send rate should be, at which they send updates to each other. If an obstacle in between the two players blocks the view on each other (you might even want to design your maps in a way that makes sure that this happens more often), then they might not need to exchanges updates with each other at all, as they can't see each other anyway.

For heavily physics based multiplayer games it is absolutely critical to run local physics simulations for remote players to only correct the local simulation results with the values that are incoming over the network (don't just set the local values to the incoming values, but smoothly approach them over many frames, to make it look good), so that you can reduce the rate at which you have to send the physics values over the network.
Yes, physics will become less accurate, but traffic consumption is exactly the reason, why you won't find real time multiplayer games with a gameplay that is heavily relying on physics, that have lots of players per room and super-accurate physics. All games that try this, either fail, or run local physics for approximations to keep their update-rates lower, or reduce the amounts of players per room.

Finally you might want to think about porting your game from PUN to Quantum, as good-looking networked physics at reasonable bandwidth consumption is one of Quantums key strengths.

PhotonUser
2021-06-18 18:49:58

@Kaiserludi wrote: »

Hi @PhotonUser.

The 3GB per CCU are based on the traffic consumption of an average Photon Cloud app: The value has been chosen so that most apps stay below it. If we would raise the amount of included traffic per CCU, then we would have to raise the price considerably. Including twice the amount of traffic would result in approximately twice the price, as traffic is the main cost driver. As most apps don't consume that much traffic, we don't include more than 3GB per CCU to be able to offer lower prices than what we otherwise could offer.

Developers, who want to consume more traffic and are OK with paying for their higher consumption, can simply subscribe to more CCU than they have, to raise their overall amount of included traffic, or they can choose to pay overuse fees for the amounts of traffic by which they surpass the amount included in their subscription.

As you will notice, if you do the math, you also surpass the recommend maximum amount of messages/s by a factor of 2.
This is not surprisingly as you are having 10 players per room and as when every player sends everything to everyone else in the same room, then the traffic and messages/s increase squared to the amount of players per room.
For this reason it is absolutely critical for games with more than 8 or so players, and when sending a lot of data per player, then even with more than maybe 4 or so players, per room, to NOT send every message to everyone else inside the same room, but to implement some form of interest management:
The further two players are apart from each other, the lower the send rate should be, at which they send updates to each other. If an obstacle in between the two players blocks the view on each other (you might even want to design your maps in a way that makes sure that this happens more often), then they might not need to exchanges updates with each other at all, as they can't see each other anyway.

For heavily physics based multiplayer games it is absolutely critical to run local physics simulations for remote players to only correct the local simulation results with the values that are incoming over the network (don't just set the local values to the incoming values, but smoothly approach them over many frames, to make it look good), so that you can reduce the rate at which you have to send the physics values over the network.
Yes, physics will become less accurate, but traffic consumption is exactly the reason, why you won't find real time multiplayer games with a gameplay that is heavily relying on physics, that have lots of players per room and super-accurate physics. All games that try this, either fail, or run local physics for approximations to keep their update-rates lower, or reduce the amounts of players per room.

Finally you might want to think about porting your game from PUN to Quantum, as good-looking networked physics at reasonable bandwidth consumption is one of Quantums key strengths.

By following the formula:

• a room has N players.
• a player sends F messages per second (message send rate in Hz)
• average message size is X (in bytes, payload (P) + protocol overhead (O))
• an average player spends H hours per month on your game

to calculate traffic per CCU per month, a 3KB/s CCU would consume:

C = X * F * N * H * 60 (minutes per hour) * 60 (seconds per minute)
C = 300(3KB / 10 msgs) * 10 * 10 * 60 * 3600
C = 6.48GB per CCU per month

I would not mind paying double the amount per CCU at all.

But if I calculate it from a server perspective consuming 300KB/s beacuse its sending players info to all players. that would give me way different results, did i calculate it incorrectly in the formula?

About the interest management, that would be done with interest groups?

Also, maybe BOLT is a better suit since it will try to connect players directly first? So one of them will act as the relay and the ones that can't connect use the cloud, that would minimize the costs a lot i guess.

Kaiserludi
2021-06-18 19:42:09

Hi @PhotonUser.

If every player sends every message to every other player inside the same room, but not to himself, then the correct formula for the overall traffic per CCU per month for a 31 day month is:

C = X * F * N² * 31 (days per month) * 24 (hours per day) * 60 (minutes per hour) * 60 (seconds per minute)
C = 300 * 10 * 10 * 31 * 24 * 60 * 60
C = 80352000000 bytes or 80GB per average CCU per month

However the amount of CCU that you subscribe to are the sum of the peak CCU per region.
So if at one day in your month the sum of the CCU-peaks of your game across all regions is at 1000, then you would need to subscribe to 1000 CCU.
The average CCU will be considerably lower than that as the amount of players will vary between different times of the day and between weekdays and weekends.

Also keep in mind that the peak CCU is bigger than the max overall CCU across all regions.
So if you only have two regions, EU and US, and at one time of the same day you have 500 CCU in EU and 100 in the US and at a different you have 100 CCU in EU and 500 in US and at a third time you have 400 CCU in EU and 400 is US, then the sum of the CCU across regions at these 3 points in time would be 600, 600 and 800, so the max CCU across all regions would be 800, but the sum of the CCU peaks would be 500 (highest value in EU) + 500 (highest value in US) == 1000, although at other times of the day you might only have like 20CCU in EU + 20 CCU in US.

So if the average CCU is 10% of the peak CCU across regions, then that would result in 8GB per peak-CCU.

PhotonUser
2021-06-18 22:46:32

@Kaiserludi Thanks for the answers.

Ok so the plans assume peak CCU, MAU is also guessed based on peak CCU? for example 100 CCU plan estimates around 100 * 20 * 20 -> 40K MAU, even though average CCU might be 20 players(or even lower), so only rarely should any game be hitting its peak, did I understand correctly? If Yes than it looks better to me. than I would just be charge some extra fees for peaks at the end of the month, sounds good.

Do you have any idea of how much data (KB/s) an average player on a fast paced game sends to photon servers?? i know it varies a lot but somewhere in the middle?

Also Would you advise me to switch over to quantum? I see its only an offer for studios, I am not on that level yet, but it would probably help me a lot.

Kaiserludi
2021-06-18 23:35:28

Yes, the MAU-estimates are also based on peak CCU.
However typically games have a peak each day, with the average being a lot lower than the peak CCU for most of the day, but with the peak CCU occurring every day.
You likely will still have considerable traffic overuse with your current amount of data per room.. In my example above you would use 8GB per peak CCU, while 3GB are included per peak CCU.

One more point that I have forgotten is:
with 10 player matches depending on your matchmaking players might wait for some time, until a room is filled and the game can start. They would still count as CCU, but barely generate any traffic, while they are online waiting and only sending keep-alive messages.
Also not all 10 players inside a room may stay until the very end of the match.
This both would reduce the monthly traffic per peak CCU compared to the theoretic scenario that all players would be actively playing 100% of their online time and everyone would always play until the end of a game.

Do you have any idea of how much data (KB/s) an average player on a fast paced game sends to photon servers?? i know it varies a lot but somewhere in the middle?

No, I don't really have any numbers at hand for this, sorry.

Also Would you advise me to switch over to quantum? I see its only an offer for studios, I am not on that level yet, but it would probably help me a lot.

Generally Quantum is target at experienced developers, not at beginners, as it has a steep learning curve.
Furthermore switching an existing project to Quantum essentially means rewriting the game code from scratch.
Also it might not be affordable to you, if you are a hobbyist.
That said I definitely think that your game would profit a lot from using Quantum.

PhotonUser
2021-06-18 23:53:20

@Kaiserludi Again thanks for your time.

I see that quantum relies on lots of decoupling, so that may not be time effective for me, atleast for development, as I use some 3rd party libraries to handle some of the physics behind my game, and from what I see Quantum has its own physics engine independent from unity where I would need to somehow plug these 3rd party libraries into it, so not really happening...

Maybe BOLT is a better solution since majority of clients are gonna be connected directly to a host(player) and some would go through the cloud. Whats your take on it?

There is also FUSION coming up...

Kaiserludi
2021-06-19 00:09:29

Hi @PhotonUser.

Bolt Pro offers p2p connections and data that goes through those p2p connections, does not count against the cloud traffic, so yes, with Bolt only traffic of players, for which NAT-punchthrough fails because of their firewalls (approximately 10-15% of players) + stuff like matchmaking (which should cause negligible amounts of traffic compared to your current per room traffic), would count against the cloud traffic.
So with a switch to Bolt Pro you could achieve a major reduction of your cloud traffic consumption by probably more than 80%.

Note that Bolt Free only offers relay-connections, so you would need to switch to the Pro-variant to profit from this, when you reach a significant user base.

Fusion also offers p2p connections, but I don't know if they are included in all pricing-options. This is probably not even decided yet, as Fusion isn't final yet.
Fusion would be an option, if you don't plan to release your game in the next couple of weeks, as it won't be released before August, maybe later.

Finally the Photon Realtime C++ SDK also offers p2p connections (independently of your subscription plan), but you would need to write your own Unity native plugin in C++ to access it from Unity, which would be a considerable amount of work on your site. You should have some experience with C++ to consider this option.

PhotonUser
2021-06-19 01:12:33

So from what I understand BOLT requires some talking to Exit Games in order to download the PRO sdk and start developing with it, the free version is just like PUN but with extra features like lag compensation and more if I understood correctly.

On photon BOLT, do hosts act as a relay for other peers and master client?

Btw im kinda confused on mixing cloud relay with BOLT p2p tech. How can some users connect to a peer and other users to the cloud relay for the same room? I think the host also connects to the cloud relay then? So it can communicate with hard firewall users.

Kaiserludi
2021-06-21 15:26:58

Hi @PhotonUser.

I have just talked with our lead Bolt developer @ramonmelo, as I am not very familiar with Bolt myself and apparently I was wrong:
Bolt Free does offer p2p connections. The difference is, that with Bolt Pro you can use those p2p connections with your own matchmaking and optionally completely without a Photon Cloud connection, while with Bolt Free you still need to be connected to Photon Cloud, as Photon Cloud is used for matchmaking, but also for the NAT-punchthrough and as a fallback relay.

the free version is just like PUN but with extra features like lag compensation and more if I understood correctly

For the Bolt feature set, please see
https://doc.photonengine.com/en-us/bolt/current/getting-started/overview

On photon BOLT, do hosts act as a relay for other peers and master client?

No.
In Bolt you have one instance that runs as a 'server', which is either a dedicated server on a machine hosted by yourself, or one of the clients.

Btw im kinda confused on mixing cloud relay with BOLT p2p tech. How can some users connect to a peer and other users to the cloud relay for the same room?

Each of the other clients attempts to establish a p2p connection with the 'server' instance. If establishing this p2p connection fails for one of those other clients, then that client will communicate with the 'server' through the Photon Cloud relay, while that 'server' will still communicate with the other clients through p2p connections.
If the 'server' instance can't establish p2p connections, then establishing p2p connections will fail for all clients in that room and everything will go through a relay. You can detect this case (simply check on the 'server instance' if all connections are relayed) and restart the procedure with a different client being selected as the 'server'.

I think the host also connects to the cloud relay then

In Bolt Free all clients are always connected to the Cloud relay. They are connected either only to the relay or to the relay AND p2p to the 'server' instance.

Bottom line:
If you use Bolt Free without a dedicated server, then it should indeed solve your Cloud traffic problem, as you would only have relevant amounts of traffic for communication between clients, between which no p2p connection could be established, which should be the minority.
With a dedicated server you would still have a lot less traffic on Photon Cloud, but you would then just have moved that traffic to your own self-hosted server and would have to pay for the traffic to your hosting provider. In case of a bare metal server you would not really have won anything, as the costs should be comparable to those on Photon Cloud. If your dedicated server would run in a Cloud (like on Amazon AWS or on Azure), then you would even be much worse off with having the traffic on that machine than with having it on Photon Cloud, as traffic on such Cloud instances is a lot more expensive than on Photon Cloud. So stay away form the 'dedicated server' option and use the 'listening server' (one of the clients is the 'server') option.

I was wrong with my 10-15% estimation, by the way. Apparently on average only 5-10% of connections are relayed (and that is even without your code detecting the case, in which the 'server' instance can't establish connections - the numbers should be even lower if you handle that case), so Cloud traffic should only less than 10% of your overall traffic, while over 90% goes through p2p connections.

Finally Bolt also has it's own compression system that may help with reducing the traffic, too, although as you appear to already use some compression of your own, I am not sure, if Bolt could compress that much more, as usually compressing already compressed data does not achieve a lot of further compression, except when the original compression was very ineffective.

PS:
For Bolt specific questions you should create a new thread in the Bolt section of the forum, so that the Bolt developers see it.

PhotonUser
2021-06-21 18:16:54

I am considering going with self hosted PUN server because I did some math and it appears to be cheaper?

Photon Cloud charges about 0.075$ (average from .05$ and .1$ per GB consumed

Assuming worst case scenario excluding players that are on lobbies searching for matches or just idling on the menu.

If we take it from the examples above for a 10 player match where the server has a 300KB/s load, it would amount to 1GB/hour/match/10CCU.

I looked at the servers from G-Core labs, for US they offer 30TB for $175

Photon charges 0.05$ per GB on US.

That would be $1500 for 30TB

Of course to really make use of that server I would need 300+ CCU at all times and would need to deal with the servers myself.

Kaiserludi
2021-06-22 16:56:24

Hi @PhotonUser.

Yes, that should indeed be cheaper in your case. I wasn't aware that G-Core is nowadays including that much traffic without extra charges.