Outgoing commands growing constantly

CousCousiere

Hi,

I am working with LoadBalancing using Photon 3.2.2.1. My test is made using 2 computers and one of them has the LoadBalancing service installed. Everything works fine for the connection : the game is launched and everyone can play. The service() function is called 10 times per frame on every computer. In the example, there 13 replicated creatures.

Each creature replicates its transformation using a non reliable buffer (24 bytes for each entity), and its states changes using a reliable buffer (states changes only happen every 4 or 5 seconds for a creature). To reduce the number of packets sent for transformation, I put all of them inside only 1 Photon::Hashtable so the transformation of all the entities uses only 1 non reliable message.

The problem is that the QueuedOutgoingCommands is slowly growing (only on the GameMaster side) from 0 to 10 or more. For information, the GameMaster is the one who manages 12 entites (the last one is managed by the computer of the client who plays this character) so I don't think that the problem comes from the GameMaster : I think that it happens on the computer that has to manage many entities.

The result on screen is that the non reliable packets sent from the GameMaster takes more and more time to be sent to the clients (is doesn't affect reliable packets). The non reliable packets sent from the clients to the GameMaster are not affected. After 7 or 8 OutgoinCommands on the stack, the packets are not received by the clients.

There are the network statistics :
- ping 103ms (service() called 10 times per second)
- Upload : 8.65Kb/s
- Download : 1.24Kb/s
- Queued incoming commands : 0
- Queued outgoing commands : 4 (and growing)
- Number of reliable packets sent : 1/s
- Number of non reliable packets sent : 10/s

Do you know if there are limits for the amount of packets / data size / anything that could explain this problem ? If I send too much data, is there a way to compress the Hashtables to reduce their size ?

Another question : I use Hashtable because I haven't found another type, but I just want to create my own buffer which is just a sequence of values that are read in the exact same order than they are written, so I don't need a Hashtable, just a binary buffer. Is there a way to have such a type ?

Thanks by advance for your help.
Chris

Kaiserludi

Hi Chris.

One service() call will just trigger one single call to sendOutgoingCommands() and sendOutgoingCommands() only sends out about 1.000 bytes of payload per call, because udp packets are limited in size (the 1kb number is just a rough estimation, the exact number of bytes can vary per packet and depends on how much overhead is needed to be added by udp and Photon for structuring - for example Photon needs to add more overhead for a Hashtable than for a byte array with the same payload size).

In your case you are sending about 8,65kb/s and do 10 service() calls per second, so that is 865byte of outgoing payload per service-call and you are sending everything as Hashtable. Depending on the structure of your hashtable (amount of key value pairs vs. size of the values) you may end up with quite a lot of overhead. Additionally as Photon doesn''t split up message, which would suit in a single udp-packet, packets may not get filled up completely (example: you have three messages of each 600 bytes in size -> two of them fit into one 1000 bytes packet, but they are small enough to fit into one on their own -> Photon won't split them up, so they will be sent in 3 packets, although their payload sum would fit into 2).

That way apparently the data that you are sending per second does not fit into 10 packets.

As a solution you could just call service() more often. To avoid unnecessary overhead for checking the incoming queues way more often then necessary you may also want to pass false for the dispatch flag to service().

Photon doesn't offer any built-in compression algorithm, but you could of course just run any compression that's avaialble on all platforms that you want to support.

As you don't need key-value mapping, it would certainly be a huge optimization, to send an array instead of a Hashtable. If your elements aren't all of the same type, then you could still use an Object-array, which would still save you from all the keys. If all elements are of the same type, then a typed array would be the better option, so that the element type information has to be sent only once for the whole array, not once per element.
The Custom Type Interface may also be an option, but as you mentioned a binary buffer, your best shot would probably be to just send your data as a byte-array.

CousCousiere

Thanks very much for your quick answer.

I'll call service() more often, but obviously it won't be enough : removing the Hashtable and replacing it by a byte-array seems to be a must have.

So, my question is : how can I use a byte-array ?

To send a message, I use the function LoadBalancingPeer::opRaiseEvent, and this function uses a Hashtable. Do you have a tutorial or any documentation explaining how to do it ?

Thanks again
Chris

Kaiserludi

In 3.2.2.1 you would just add a single key value pair to the Hashtable, with the value being the byte array.

From 3.2.3.0 upwards opRaiseEvent() also directly accepts any (of the supported by photon ones) type of data as parameter, not just Hashtables.

CousCousiere

OK, I'll update the Photon version to use the new version of opRaiseEvent().

Thank you very much !
Chris

CousCousiere

I've updated Photon to the latest version and I've modified my buffer to be an array of nByte.
It does work well for opRaiseEvent, but I can't understand which function to use to receive this buffer on the other clients.

Previously I used customEventAction (and maybe it's the same now) but I just can't understant how to extract my nByte array from "ExitGames::Common::Object& eventContent".

Could you help me on that ?

Thanks by advance
Chris

Kaiserludi

Hi Chris.

It's still that same function.

You can get a ValueObject<nByte*> from that Object this way:
[code2=cpp]ExitGames::Common::ValueObject<nByte*>(eventContent);[/code2]

Afterwards you can call getDataCopy() or getDataAddress() on that ValueObject, to access the nByte* or a pointer to it (which would be a nByte**) and getSize() to get the element count of that nByte*.

If you use getDataAddress(), then the return value will only be valid as long as you stay in the scope in which the ValueObject instance exists. So if you need it later, you will have to deep copy it yourself.
If you use getDataCopy(), then the whole array will be copied into newly allocated memory. You will have to deallocate that memory, when you are done with it. The correct way to deallocate it is passing it to ExitGames::Common::MemoryManagement::deallocateArray().

You can find this info by having a look into the demo and into the API reference for the mentioned classes and functions.

CousCousiere

Hi and thanks for your answer, it seems to wompile.
Now I have a problem with the new photon version : even if I send multiple opRaiseEvent, nothing is received anymore on the client with customEventAction.

Is there something to do with the anthentification ?

Thanks for your help and sorry for all my questions.
Chris

Kaiserludi

Phew. That can have a lot of causes.
Please have a look at all 3 of the cpp release history files (for Common, Photon and LoadBalancing) inside the SDKs doc folder for changes since your old version. That may give a hint in which parts of your code you have not yet made some needed adjustment to the version update.
Looking at differences between the demo and your code and also between the new and the old version of the demo might also help in what you have to adjust.

This entry from the loadBalancing release history (paragraph for 3.2.3.0) might be related:

- changed: Peer::opRaiseEvent() doesn't provide different overloads for targetActorNumbers and eventCaching + ReceiverGroup anymore, but the first one doesn't rule out the other 2 anymore and vice versa, so if you are using these parameters, please adjust your calls to the changes in the parameter order

CousCousiere

OK I'll take a look at this tonight.
Thanks for your answer.
Chris

CousCousiere

I found the problem but not yet the solution : buffers are sent if they are Hashtables, but it seems that the way I try to send my newly created nByte* buffer doesn't work. I haven't found any example in the demos and the documentation didn't help, that's why I ask you.

I call the function this way (buf : nByte* , size : unsigned int , type : int)
mLoadBalancingClient.opRaiseEvent<nByte*>(false,buf,size,type);

I have tried to declare a ValueObject<nByte*> this way, but it doesn't compile :
ExitGames::Common::ValueObject<nByte*> obj(buf,size);
mLoadBalancingClient.opRaiseEvent(false,obj,type);

It seems that it's just a syntax problem but I really have troubles to understand the opRaiseEvent declaration, that's why I am a bit stuck... sorry for that.

Kaiserludi

Hi Chris.

The declaration may indeed look a bit confusing, but the usage actually is pretty straight forward:
[code2=cpp]const short ARR_SIZE = 10;
nByte arr[ARR_SIZE];
for(short i=0; i<ARR_SIZE; ++i)
arr = i;
mLoadBalancingClient.opRaiseEvent(reliabilityFlag, arr, ARR_SIZE, eventCode);[/code2]

CousCousiere

OK so I really have a problem because your code doesn't raise any event on the client side (but a Hashtable does work).

Is this feature available since a long time ?
My server version is 3.2.6.2037, I've updated it this morning.
It's PhotonServer, not PhotonCloud.

Because a hashtable works, I'm trying to fill a Hashtable with my buffer like this :

const short ARR_SIZE = 10;
nByte arr[ARR_SIZE];
for(short i=0; i<ARR_SIZE; ++i)
arr = i;

ExitGames::Common::Hashtable table;
table.put<nByte>(0,arr,ARR_SIZE); OR table.put<nByte*>(0,arr,ARR_SIZE); OR table.put(0,arr,ARR_SIZE);
mLoadBalancingClient.opRaiseEvent<ExitGames::Common::Hashtable>(false, table, 0);

I don't know if it's the right definition but, even if I receive an event on the client side, I can't find a way to read it.
I tried this :
ExitGames::Common::ValueObject<ExitGames::Common::Hashtable> tmp = eventContent;
ExitGames::Common::Hashtable table = tmp.getDataCopy();
nByte* array = (ExitGames::Common::ValueObject<nByte*>(table.getValue(0))).getDataCopy();
=> array = NULL

Really sorry to bother you with my questions, I think that I am just missing something in the type definition.

Kaiserludi

Hi again.

I have just tested with the current server SDK release and have just realized that the feature to directly send any kind of data without first wrapping it into a Hashtable is not available with the current Photon Server SDK version. Its available with Photon Cloud and will be part of the next Photon Server release. One reason more to bug my colleagues to release a new Photon Server SDK.

About your attempts to wrap the nByte* inside a Hashtable:
Your first attempt can't work, because you are sending the key as a nByte, but are looking for an int key on the receiving side. A key of type int and value 0 and a key of type nByte and value 0 are two completely different keys, that can happily co-exist in the same Hashtable and don't rule each other out. It's the same like if you were putting an entry with key "1" into an empty hash and then try to read out an entry with key "2" - you won't find that entry.
Your second attempt tries to add a key of type nByte* and value 0 (meaning a NULL pointer to a nByte array). That's not supported and Photon actively makes the compiler refusing from even compiling that to prevent more difficult to spot run time errors.
Your third attempt table.put(0,arr,ARR_SIZE) works just fine for me in combination with your readout code on the receiving side, however it would still use int as key type and using 4 bytes for just a dummy key, although nByte would be totally sufficient and would save 3 bytes. Sure, 3 bytes are just a minor optimization, but if you are always using the smallest type that is sufficient for all your keys and values everywhere, especially for data that gets sent multiple times a second and to multiple receivers, then the savings sum up and you may see a significant influence on your traffic costs (realtime gaming produces a LOT of traffic - on the Photon Cloud the average game server machine handles about 1,000-2,000 CCU and produces 6.5-13 Tera(!)bytes of traffic per month).

Therefor I would do it in the following way:
[code2=cpp]const short ARR_SIZE = 10;
nByte arr[ARR_SIZE];
for(short i=0; i<ARR_SIZE; ++i)
arr = i;
ExitGames::Common::Hashtable hash;
hash.put(static_cast<nByte>(0), arr, ARR_SIZE);
mLoadBalancingClient.opRaiseEvent(false, hash, 0);[/code2]

[code2=cpp]ExitGames::Common::ValueObject<ExitGames::Common::Hashtable> tmp = eventContent;
ExitGames::Common::Hashtable table = tmp.getDataCopy();
nByte* array = (ExitGames::Common::ValueObject<nByte*>(table.getValue(static_cast<nByte>(0)))).getDataCopy();
// do something with that received array here
ExitGames::Common::MemoryManagement::deallocateArray(array);[/code2]

CousCousiere

Hi,

Thanks for your answer : it's really good to hear that I am not becoming crazy !
And you are right : it's far better to use the smallest key size possible, and nByte seems to be the better choice.

I've done what you have suggested and everything works now !

Thanks a lot
Chris

CousCousiere

Hi and sorry to revive this old post,

I've updated to the latest version (client & server) of Photon and I hoped that sending directly a nByte array would work now.
I don't now if you remember our conversation but I wanted to send a nByte array by calling :

const short ARR_SIZE = 10;
nByte arr[ARR_SIZE];
for(short i=0; i<ARR_SIZE; ++i)
    arr[i] = i;
mLoadBalancingClient.opRaiseEvent(reliabilityFlag, arr, ARR_SIZE, eventCode);

but ut didn't work with a previous version of Photon.
I had to store my nByte array into a Hashtable to make it work, but it was supposed to be temporary.

And it seems that it's still not working with the latest version... could you confirm that it doesn't work yet ?

Thanks by advance
Chris

Kaiserludi

Hi Chris.

I am sorry, but I can't confirm this. It works like a charm for me with your provided code and v3.2.5.3.

Please make sure that you are also using an up to date server version, as older server versions do not support this.

CousCousiere

Hi kaiserludi,

The problem is that nothing is received on the other machines when calling this code, but it compiles well : do you receive something on your side or did you just try to compile the code ? I've updated to server version 3-4-22-6309 and client to 3.2.5.3.

CousCousiere

Well, it seems that I failed updating the server : I removed everything and reinstaled it, and it works well now...
Deeply sorry for that and thank you for your help !

Chris

Kaiserludi

Thanks for the update. I am glad it works for you now :-)