Synchronization in Loadbalancer Application
Options
jashan
✭✭
As I'm currently extending the loadbalancer app with my own logic, I was wondering if I need to synchronize the data between the different game servers?
In particular, I'm adding game groups and teams which have actors in them. I'll be using rooms to represent the game server connections. In other words: Each game server (Unity based game servers) will be the MasterClient in one room. Each game server can host multiple game sessions at the same time.
I use game groups to represent game sessions (i.e. players playing together). Using rooms for that doesn't sound like an option because then I'd have to make sure that one server can connect to multiple rooms.
So, what I'll be doing is have game groups, and in those game groups players can form teams. When all players that are in teams (there could be some in the game group which are not in a team) decide to start playing, the game group with its players is sent to the Unity game server which "hosts" the actual game (i.e. server-side logic for the game; connection is still done via Photon and I might gradually move game logic from the Unity server to Photon where it's easy to do).
So, the questions are: If Actors end up in different instances (Game1, Game2 etc.), will they be able to be in the same "room"? And if a game group exists in one instance, would it have to also exist in another instance? So, do I have to make sure all actors, rooms, game groups and teams always exist on all instances (Game1, Game2, etc.), or would I have to make sure that if a player wants to join a game group that is not available on his current instance, he is transferred to the instance where the game group exists?
In particular, I'm adding game groups and teams which have actors in them. I'll be using rooms to represent the game server connections. In other words: Each game server (Unity based game servers) will be the MasterClient in one room. Each game server can host multiple game sessions at the same time.
I use game groups to represent game sessions (i.e. players playing together). Using rooms for that doesn't sound like an option because then I'd have to make sure that one server can connect to multiple rooms.
So, what I'll be doing is have game groups, and in those game groups players can form teams. When all players that are in teams (there could be some in the game group which are not in a team) decide to start playing, the game group with its players is sent to the Unity game server which "hosts" the actual game (i.e. server-side logic for the game; connection is still done via Photon and I might gradually move game logic from the Unity server to Photon where it's easy to do).
So, the questions are: If Actors end up in different instances (Game1, Game2 etc.), will they be able to be in the same "room"? And if a game group exists in one instance, would it have to also exist in another instance? So, do I have to make sure all actors, rooms, game groups and teams always exist on all instances (Game1, Game2, etc.), or would I have to make sure that if a player wants to join a game group that is not available on his current instance, he is transferred to the instance where the game group exists?
0
Comments
-
All players of a game instance should connect to the same server. The load balancing app assigns a new server just for new games, all players that join the same game (with same name) get the same server address returned - unless you changed that.0
-
Oh ... that's good to know ;-) ... so then I'd probably be better off using the Lite application instead of Loadbalancer. After all, this is only one single game.
Does Photon Unity Networking work with Lite as well as it does with Loadbalancer or is there anything I'll need to change for that to work?
On the other hand ... in case I ever need to support multiple Photon servers, it might be easier to extend Loadbalancer to handle this situation than extending Lite ... right?0 -
the game server part is a Lite extension, so in general you should be able to get PUN to work with Lite. You have to rip out all the master server stuff though.
that's exactly why PUN comes with the loadbalancer project, and not just Lite.jashan wrote:On the other hand ... in case I ever need to support multiple Photon servers, it might be easier to extend Loadbalancer to handle this situation than extending Lite ... right?0 -
Well, we should really add a check into the forum-software, which prevents one from making posts, which include the word "game", it is just too confusing, when used with Photon.jashan wrote:Oh ... that's good to know ;-) ... so then I'd probably be better off using the Lite application instead of Loadbalancer. After all, this is only one single game.
So, do you mean, you have just one application (like for example Starcraft vs. Warcraft) or actually just one single room (meaning not like in the previously mentioned examples a real lot of open matches at the same time, but more like in WOW, one big world for all you players)?
If you have just one room, then LoadBalancing will not gain much for you, but then the whole room-based concept from Lite, which is optimized for many rooms with not too big amounts of players per room (for example per default raiseEvent() will trigger an event for everyone else in the room, which will cause a real lot of traffic, if you have thousands of players in a single room, all calling raiseEvent() without specifying receivers) is probably not the best for you kind of game, but the concept from MMO would suit you much better.
If you have multiple rooms, then LoadBalancing will deal them out to different game servers.0 -
It's one application that acts as a "single game" on some levels, and as "many game sessions" (like many rooms) on others.
I guess the tricky and confusing part is that it's kind of a mix: I have players that are free to jump from one game group to the other (while not playing). So that part is "strictly room-based" and from that perspective, Lite would be perfect, and "game group" would be equal to "room". If I have more players than a single Photon server can take and I'd use Loadbalancer, I'd somehow need to find a way to "transfer" users from one instance to the other in the case that they join a game group on that other instance. Like, you could come online and look for one of your friends and join the game group they are in. Or you want to join a game group that's currently playing a specific level. So you need to be able to see and join all currently existing game groups (even if they were spread across different instances).
Also, I might have to transfer a whole game group with its players from one instance to the other (e.g. if there's much less game groups on one instance than on the other). Game groups can be joined when they are playing a session - but you don't join the session (instead you have to wait until the session is completed to join a team, then when the next session starts, you're in).
This could probably work with Loadbalancer but might need quite some tweaking. To keep things simple, I'd rather not worry too much about these "server transfers" for now - but I'd like to make sure that I can implement them later on when the need arises, without having to pull everything apart ;-)
That was the first part which is more a long-term consideration ... the second (near-term) is: "Behind" Photon, I have an authoritative Unity game server that could host game sessions for multiple "rooms" simultaneously. I'd expect to have a few of those servers connected to a single Photon server. And it seems the best way to achieve that with least effort is have "one Photon Room = one Unity game server". That would be 100, maybe 200 players assigned to one Unity game server (I need further testing to figure out how many exactly).
So, in my case, "CreateRoom" will be the only way to create a room, and this is only allowed to be done by MasterClients (which really are my Unity game servers). JoinRoom is only for clients, and clients can only join rooms that were previously created by the Unity game servers (no implicit room creation). It's like an internal loadbalancing: When a game group starts a game session, all players from the group might first be moved into another room (i.e. moved to another Unity game server) if the one they are currently in says "no more, please".
Speaking of traffic: I'll need a way to have raiseEvent() only send the events to the players of one single team, to the players in a game group that are actually playing in a level, to all players in a game group (including those not currently playing and waiting for the next session to start), and to all players currently online. And of course, all players currently playing in a level need to be able to send messages to "their" Unity game server (=MasterClient). As the Unity game server keeps track of which players belong to which game group (and a couple of other things ;-) ), it can have multiple game groups playing simultaneously (in different levels, or even in the same level), without interfering with each other.0 -
I would propose to open a separate room for every game group, which would work like a kind of sub-lobby, where all the current members of that group, that are currently not playing, would be in.
Then you should open a separate room for each match, which is currently played by members of that group.
So for example group one has 1 sublobby-room for the not currently active players, where they could chat, could get updates about the group, etc., but no events, which would only be relevant ingame, so that here there is no problem in having 200 players or so in one room.
Then there would for exampke be 20 game rooms open for that group, each with on average 10 players, who would receive the ingame-data of that specific match.
You should not really care about the question, on which game-server which room is - this is the task of the LoadBalancing app and it even does not matter, if a game room is on another game-server then the group-lobby for the players of that match.
Alternatively you could of course handle the complete group as one big room, but then you would have to track yourself, which actorID currently is in which of your matches, so that you would only add those acotrIDs to the receivers-list of a raise-event with ingame data in it, as Photon could not track the matches, if they do not have separate rooms. So you would have to do more tracking yourself. Well, you will need to organize receiver-groups anyway to manage sending of "for my tam and not for opponents only" messages and that kind of stuff, but with this appraoch of just one rrom for all matches of one group you would additionally have to distinguish, if join- and leave-events are relevant for you, or not and perhaps have to modify the serverside code, to only send them to clients in currently the same match, in case your groups can get that big, that sending joins and leaves do all group-members will add to much traffic.
Moreover with the second approach you will need a few more game-servers than with the first one, even if you optimize it by server-side modfications described above to not produce ore traffic than the first approach: If a specific game server could handle 980 players (when due to optimizations it is nor relvant, how many of them are in the same room, as you do not send everything to everybody in the room, otherwise the difference would be far higher, of course), then it could only handle 4 rooms with 200 each, making a sum of 800, but it could handle 49 rooms of 20, making a sum of 9the full 80 (as explained above, the group-lobbies would not have to be on the same game-server like the actual matches, so if a game-server can handle the matches for 4,9 groups, it can really do this and not only handle the amount floored down to the matches for 4,9).
On the other hand the second approach would provide you with the advantage, that you could easily send data to players in different matches of the same group with built-in functionality and have not add the by far not trivial functionlity of exchanging this stuff between different rooms on different gameservers yourself, in case, you absolutely need this, as thios is currently not provided by Photon.
As a side note:
So, you have a Photon master server with quite a few game-servers, lets say, 100 of them, then you have multiple Unity serverside instances per Photon game-server, lets say, 20 for each, next you have multiple groups per unity-serverside instance, lets again say 20 for each, and then you have mltiple active matches per group, lets stay with 20, and finally you have multiple players per match, to guess a number, lets say 20 again.
This will end up with 100x20x20x20x20 == 16 million active players at the same time, not even counting the ones, that are oline, but not actively playing.
Well, even the whole Battle.net, which is one of the biggest game-net, has "only" between 500k and 1 millon users online at the same time most of the time and that number is including the users, that are not currently playing, but just online and it is for all the Blizzard games together, not only one of them.
So, perhaps my numbers have been to big.
Lets make another try:
20x10x10x10x10==200.000.
Still 200k currently active players and even more currently online players, which would still mean, one of the biggest online-games ever.
Of course, I could reduce the numbers even more, but if I drop them even under ten per level, then I have to ask myself, if it would not be a bit to many different kinds of spreading for the absoulte amount of players and therefor a uneeded level of complexity.
Are you really sure, that it is realistic for your game to get that huge amounts of players, onlien at the same time, which would mean millions of players overall, in the first few month after going live? If not, than you should probably think about all that abstraction when and if you (ever) see, that you will really need it soon.0 -
Kaiserludi wrote:I would propose to open a separate room for every game group, which would work like a kind of sub-lobby, where all the current members of that group, that are currently not playing, would be in.
That sounds cool. How can I make one Unity game server using PUN connect to multiple rooms and be the MasterClient for each of those rooms?
If I can solve that easily, I'm all happy with having "room == game group" ;-) I just need to make sure that I can have "one Unity game server can handle multiple game groups at the same time". With Unity's networking, this is already implemented and working fine - I just need to get this into Photon / PUN in a way that works well.
So, in that set up, I'd have a maximum of 24 active players per room because "room == game group". A game group consists of up to four teams, with a maximum of 6 players per team, hence 24. There could be more players in the game group (actually, I don't limit the number) - but they cannot participate in the action, so I don't worry too much about those (even if they were "spectators" of the current session, they wouldn't touch the Unity game servers at all - my Photon server would simply distribute the messages to them, which I believe only will become a problem if it's a really large group of people passively watching ... maybe some day ... maybe not ;-) ).
The only reason I'm considering having "Unity game server == room != game group" is because it seems that PUN assumes there is only one MasterClient per Room, and each master client can only join one room at a time. It's probably a good assumption for most games - but I'd really like to avoid having to instantiate one Unity game server for every game session (or at least: for every game group).Kaiserludi wrote:This will end up with 100x20x20x20x20 == 16 million active players at the same time, not even counting the ones, that are oline, but not actively playing.
I think my math is actually much simpler than that: I'll start with 4 actual servers (hardware), one handling the database, one handling Photon, and two handling Unity game servers. Actually, initially I might even start with 2: One with the database, and one handling Photon + Unity game servers.
As Unity currently is mostly single-threaded it only uses one CPU / core at a time. So IMHO, it doesn't really make sense to have more Unity game servers instances running per machine than I have available CPUs / cores. As the Unity game server already is optimized for "many game groups playing independently of each other", I'd rather not incur the overhead of starting more instances than necessary to make best use of the hardware.
The way it's set up, I could very easily add machines for Unity game servers when I need to. And that's where I most likely will need to add machines because while Photon is super-optimized to handle gazillions of players, Unity is not. With Unity's built in networking, it already got unpleasant at around 80 players on one server. Unfortunately I don't know precisely whether that was a problem with Unity's networking or the game engine itself, or both, or the specific combination of the two.
One thing I know for sure is that there were bugs in Unity's networking that did cause unpleasant behavior (NetworkTime going wild trashing the prediction which resulted in players "jumping" into nowhere). Plus: With Unity's built-in networking, I would have needed a completely separate layer to add additional game servers.
These two problems are solved with Photon.
So the next step is create a set up based on Photon which I can monitor closely during a load-test. My Unity game server has performance counters built-in, Photon has performance counters built-in, and I have a loadtester client which can simulate up to 24 players per instance (with only one networking connection, though, so this puts stress on the Unity game server much earlier than on Photon). Using PUN, I could verify that in principle, my game works with Photon - but currently I can't host multiple game sessions on a single Unity game server, and the loadtester clients also don't work; so that's why I need to do some work before I can do any actual tests.
Another thing is that the Unity game servers currently not only handle the game logic but all the handling of players, game groups, networking ("who gets which message") and persistence (= database connection, including authentication and some logging). All of that has to move into Photon because it just bloats the Unity game servers.
Speaking of numbers of players: I'll consider Traces of Illumination quite a success if I have 1,000 players online at the same time. However, I've already seen systems I was responsible for go down under load - and that's really not an experience I'd like to repeat. And having to fear that your system might crash under load is not exactly the right attitude to do any serious marketing ;-)
So if I have 800 players playing at the same time, I want to be able to scale up easily to 2,000; and when I actually have 1,800 I better am prepared to cope with 4,000. Renting a few more servers is very easy and I'm sure with a friendly talk with the provider, it's a matter of a few hours or less to set everything up. Changing the architecture when you found out "oooops, picked the wrong design" is tremendously expensive on all levels.
That's why maybe, building my Photon server on top of the Loadbalancer is still a good idea even though I may not currently make full use of it. I really don't see myself needing multiple Photon instances any time soon, assuming that one Photon instance can easily handle thousands of players. But building on top of Loadbalancer still seems smarter than building on top of Lite because building on top of Lite and then some day realizing "oh, now I need the Loadbalancer stuff" doesn't sound like fun.
Unless, of course, you say, building on top of Lite is much much easier than building on top of Loadbalancer.0 -
Lite and Loadbalancing are quite similar since the game server part of Loadbalancing is lite + the following extensions:
- a game can be marked as open or not open to deny subsequent join requests
- a game can have a max number of players
- a game can set a whether it's visible in the lobby (master server) or not
The master server replaces litelobby (litelobby is a lobby system + rooms in a single server, the loadbalancing master server can run on a separate machine from the game servers) and allows filtering by certain game properties.
2000 players for a master server should not be a problem, but 2000 players with 24 players per game might be a problem for a single server to handle - so yes, I think it's a good idea to go with loadbalancing instead of litelobby.0 -
Ok, cool ... so I'll go with Loadbalancing (that's nice because I already have my project set up and it wouldn't be much fun to now set everything up for Lite instead). So the only question that remains before I can really get started:
Is there an easy way to get one MasterClient to become MasterClient for several rooms? I don't think so but maybe I'm missing something, that's why I'm asking.
If there's no easy way, I'm pretty sure that it will be easier to port the logic I already have for players, game groups and teams over into my Photon server and handle it somewhat independently of rooms. Maybe I could make ActorNr "unique across the whole server" (instead of "unique in room") and then easily push players to different rooms server side when I need to. The clients might not even have to know which room they are in because to the client, only the game group really matters.
I'm assuming changing the logic to send events to players in game groups instead of players in rooms shouldn't be too hard ... but I haven't looked that deeply into the code so I hope I'm not up for a nasty surprise ;-)0 -
Boris wrote:Lite and Loadbalancing are quite similar since the game server part of Loadbalancing is lite + the following extensions:
- a game can be marked as open or not open to deny subsequent join requests
- a game can have a max number of players
- a game can set a whether it's visible in the lobby (master server) or not
Am I right to assume that PUN is lacking all these things at the time? (it exposes the values but they are no properties and it will never sync them back if you change them)0 -
I have to look into it to say for sure.
In theory the only thing you have to do is to set room properties with the "GameParameter" keys (as byte) - you can currently find this enumeration in namespace Photon.LoadBalancing.Operations.
You can set the following properties:
MaxPlayer = byte
IsVisible = bool
IsOpen = bool
And you should then receive these properties from the master server - plus
Removed = bool
PlayerCount = byte
We are currently cleaning up the load balancing code, so the GameParameter values might change here (255 and decreasing instead of 1 and increasing).0 -
I didn't read the whole thread (yet).
Jashan, your "only question that remains":
MasterClients are only member in one room - never in multiple ones.
We separated rooms into their own threads and made sure they don't interact with each other. All communication is done via messages, to decouple rooms and peers from each other and across classes.
You would have to rewrite the server or client for your case.
Dreamora, you are right. The properties are not yet exposed while in a room.
PhotonNetwork has a CreateRoom method with those settings. We are working on an update this week, so maybe early next week you could get it from the Asset Store.0 -
Thanks for the udpate Tobias.
I'm looking forward to it.
In case you have not decided on how to approach it: I would favor it if open etc became c# properties which upon set automatically update the corresponding Photon Property0
