gaming system for 200.000 CCU

Sergiy

Hello

I'm planning a gaming system that should be able to handle over 200.000 concurrent connections. The scheme is like this:

1. All clients connect to master game server
2. Regarding of which game client chooses, he will be transferred to appropriate game server. All game logic will be processed by this game server.
3. All traffic between client and game server passes through master server.

I'd like to ask you several questions and have a little consultation from persons who have more experience then me, please

1. Will Photon server be able to handle 200.000 parallel connections e.g send events to such number of peers?
2. How many cores and RAM should master server have to handle these connections without delays?
3. Should master server have several network cards?
4. Can single Windows Server handle this amount of concurrent connections?

thanks
Sergiy

dreamora

I'm going by the assumption that you create a game thats focused on Photons strength ie realtime, action oriented networking or similar:

The hardware required for your usage does not exist on earth where a single machine can act as proxy for that many users with realtime communication. Thats several hundred Ghz you would need to handle it.
To make matters worse you would also need networking technology that either does not exist or is extremely expensive (quadriband / infiniband, costs several hundred to thousand bucks per meter of cable) and which is not used in public clusters where you could rent space at all (amazon, rackspace etc normally use multiple gbit lines which are known to at best get 50-60mb/s bandwidth out of their theoretical 125mb/s)
Just several network cards would not remotely cut it here, you are looking at gigabytes of traffic per second due to the player -> master -> game server -> master -> player messaging pattern.
The only thing where you could nowadays get hardware to facilitate that would be the RAM side but thats about it ...

As such no, a single machine does not stand the slightest chance to handle that.

To give you an idea: Eve Online, the MMO with the highest real CCU in a single world (55k at its highest) has a 1Thz processing power cluster to handle the load and EVE Onlines 'realtime experience' is not remotely as realtime as what you might be doing and for what photon is normally used

Other games like World of Tanks or League of Legends with up to 1M CCUs use massive clusters where users do never pass through a single node and alike. They are passed around within the cluster so the cluster is loadbalanced (World of Tanks uses BigWorlds patented server technology for this purpose for example) which ensures that all have a good experience and that the hardware is optimally used. (this is more and more often handled even through virtual nodes running in VMs to utilize the hw to its max and reduce the mainteinance efforts for updating etc as you can just update the image and push it out)

Thats the reason why Photons LoadBalancing uses the master as initial node and then hands you over to the game server, to allow you to realistically scale to a massive number of game servers (as users will primarily be there, not on the master)

Philip

> 3. All traffic between client and game server passes through master server.

What is the reason for this requirement?

We've had customers with an architecture where they would do something similar only that they had several front end servers proxying all traffic to the backend servers.

- you will always have a higher rtt as without the proxy
- it works assuming your game is ok with higher rtt
- you can do it with photon
- it is hard to do (harder to do without photon )
- it is more expensive to develop and maintain

Sergiy

Thanks, Dreamora

I was afraid of that
Passing client to game server has its disadvantages. For example:
1. Client A creates a game on game server 1
2. Client B creates a game on game server 2.
3. Client C wants to play with client A and also, separately, with client B.

Now what should be done here? Client C should use 2 peers : one for playing with A and second - with B. But that does not seem to be an option . What can you recommend?

dreamora

That sounds more like you talk about a different game server than I / LoadBalancing unless you are talking about being in 2 realtime games at the same time which sounds unrealistic / unplayable. Even if it weren't realtime I don't see it possible to focus on 2 things in parallel, you normally do a thing, then switch to the next game and do a thing (asyncronous multiplayer). For such a case you would potentially handle it totally different.

If you mean that you want to start a game with A,B,C in then the problem here might be that you consider them to be on the game server even when not in a game.
In such a case, at least in case of Photon LoadBalancing / Cloud, you would be on the master server. But the moment you start the game, all the 3 of you would be pushed to the same game server into the same room to play together.

Kaiserludi

Sergiy wrote:

Now what should be done here? Client C should use 2 peers : one for playing with A and second - with B. But that does not seem to be an option . What can you recommend?

Why shouldn't that be an option? I would recommend to do exactly that: Let the client use a separate peer for every game room, that it wants to be in simultaneously.

Sergiy

My guess was because client should create a separate PhotonPeer instance for each game room. Quantity of game rooms can be 5, 10, 20, etc. Will single flash client be able to handle, say, 30 photon peer connections?

Kaiserludi

Sergiy wrote:

My guess was because client should create a separate PhotonPeer instance for each game room. Quantity of game rooms can be 5, 10, 20, etc. Will single flash client be able to handle, say, 30 photon peer connections?

Why would you need 30 parallel game rooms to be joined? There is no way, any being on earth could play 30 games in parallel anyway, so like Dreamora already mentioned, that player would concentrate on room a, then on room B, then back on room A, but not both in parallel. So why not just join a room, when an action is taken there and leave it, when another room should be joined?

Aside from that:
A client can have up to 32767 peers. It doesn't really matter that much, if you are sending/receiving data for multiple peers or for only one peer. The size and amount of the data is the relevant number and that can be to much for a client to handle with 1 peer as hard and as easy as with 30 peers.

However I don't know right know, if the Flash Client supports multiple parallel peers. I only know, that the C#, C++, objC and Java clients do and that the Javascript ones don't.

Sergiy

well, imagine a Backgammon server, or Poker server, or Checkers, etc. One can play on 20 tables at once, i have met with such people . So one of these guys can request to create game on GameServer1, while playing on GameServer2 and joining game on GameServer3.....

Can you please find out for me whether Flash client can handle multiple PhotonPeers?

Kaiserludi

Excellent example. I have played online poker at a semi-professional level myself for some time with up to 16 simultaneously open tables and I have heard of people, playing up to 100 of them and yes, it makes sense, to let the player connected to all tables at once as there is oten something going on on different table within the same 10 or so ms.
Poker Stars actually has CCU numbers around 200.000, but since Full Tilt is down, they have around 80% or so market share. You would already be the number 2 in the poker market with around 20.000 CCU, so this seems to be a more realistic number to talk about.

Still I think, multiple peers per player would be the far superior approach over routing everything through a bottle-net.
Just be aware, that every peer counts as a separate CCU (however, if you really have averages of more than 10 peers per client and 200.000 concurrently online clients, I am sure, we can make a special deal, that is accounting for he fact, that you have so many peers per client).

I doubt, that Flash is a good idea for developing the client.
As I have told you, I have experience with massive multi-tableing in online poker and I know a real lot of other multi-tablers and all agree in one thing: PokerStars and Full Tilt are the online two on the market with a software, that is suitable for multi tabling and thats THE main reason, why they are / have been dominating the market. The clients of all other online-poker providers are just getting that performance-hungry with double-digit table counts, that it gets unplayable slow and laggy even with high end pc hardware.
These clients are the kind of software, for which you have to rule out Java and even C# for performance reasons, let alone Flash, which is famous for not exactly being performant .

Sergiy

How about Unity? Can it handle 15-20 concurrent peer connections without noticeable performance loss? Can Unity achieve this as a web-browser plugin as well? AFAIK, Unity applications are running in a single-thread mode.

Kaiserludi

Sergiy wrote:

How about Unity? Can it handle 15-20 concurrent peer connections without noticeable performance loss? Can Unity achieve this as a web-browser plugin as well? AFAIK, Unity applications are running in a single-thread mode.

If you would go with unity, then you would want to go with standalone, not with web player, not because of the networking, but because one really doesn't want to play 30 or so open tables in a webbrowser, but in a dedicated application.

However I have no idea, if you can have multiple application windows with unity without a source license. Having 1 window per open game room, so that a player can freely rearrange, resize, stack, tile or cascade the tables to suit his needs best, is a must-have, if you want to address those power-players, that are playing so many concurrent matches at the same time.

Aside from that, yes, the Photon Unity clients support multiple concurrently connected peers and they also do so in the Webplayer. The peer amount shouldn't noticeable affect performance. The overall amount of data sent and received and of course non networking stuff like the graphics of 15-20 matches are more interesting. You will probably have to stay with 2D as I for example don't know any poker client on the market, that offers 3D graphics and still has an acceptable performance with more than 4 or so open tables.
The unity engine is working single threaded, but the games, using it, can still be multithreaded, even in the webplayer, so you could open additional threads for the Photon peers.

xzlxt720

I'm interesting when there're a masterserver, gameserverA,gamserverB,gameserverC,clientA,clientB, clientC.
at start, via masterserver ,clientA connected gameserverA, clientB conneted gameserverB, clientC connected gameserverC, clientA create a gameroomA on gameserverA.And then,
clientB and clientC want to join gameroomA, so they're connect to gameserverA and join gameRoomA, and gameRoomA start game logic on gameserverA.
so my question maybe , if clientB connect to gameserverB, gameserverB get info of clientB from database, then clientB connect to gameserverA, gameserverA get info of clientB from database too ? or from gameserverA ? How about update info? from gameserverA to database or gameserverB to database?

Kaiserludi

@xzlxt720:
What you have described, is not the way, how the Photon LoadBalancing is working.
Its working this way:
All clients connected to the master. Client A sends a request to create gameroomA the master server. The server responds with the address of gameserver A. Client A now switches from the masterserver to gameserver A and creates gameroomA there.
Cliebts B and C now send a request to the masterserver, that they want to join gameRoomA. The master now answers with the address of gameserver A, where it has sent client A to to create the room. Clients B and C now switch from the master to gameserver A and join gameroomA there.
The gameservers update the master in short intervals about the rooms, that are running on them, the properties of the rooms (including if a room is open or visible, the maxPlayers of the room, etc.) and about how many players are in them , so if a client that has sent from the server to a certain game server, won't ever reach it, or gets disconnected later after arriving there, the master will get this info with the next update from the according gameserver.

Sergiy

Speaking about Flash unable to handle several (10+) parallel peers - consider the situation when each peer is created by a separate flash application - new browser tab:

1. Client opens connection to Lobby
2. Client requests joining to a particular table, web browser opens a new tab with a new peer
3. Client requests joining to another table, web browser opens a new tab with a new peer
4. .......................................................

in this scenario, each flash instance has only one peer. Do you think that Flash performance will still suffer?

Kaiserludi

Depending on the web browser, all tabs of a window will still run on the same CPU core (this is for example true for even the most recent firefox versions, so that won't improve performance. I am really not so much worried of the Photon Flash clients here, but more of the performance of Flash in general, when running 10, 20 or even 30 tabs with Flash.

Aside from that as a player I would not like the idea of a tab per table, as I would have to switch through the tabs all the time, to look, at which table its my turn again. This massively decreases the amount of tables, that one can concurrently play.