Array Serialization Limit

amiractivate

Is there a limit to the size of an array that photon can handle/serialize? I am currently having issues when trying to retrieve arrays of doubles particularly. I get this error when trying to retrieve array of doubles with the size(lenght) of 20000 and above.

System.ArgumentOutOfRangeException: Argument is out of range.
at (wrapper managed-to-native) System.Array:CreateInstanceImpl (System.Type,int[],int[])
at System.Array.CreateInstance (System.Type elementType, System.Int32[] lengths) [0x00000] in <filename unknown>:0
at System.Array.CreateInstance (System.Type elementType, Int32 length) [0x00000] in <filename unknown>:0
at ExitGames.Client.Photon.Protocol.createArrayByType (Byte arrayType, Int16 length) [0x00000] in <filename unknown>:0
at ExitGames.Client.Photon.Protocol.deserializeArray (System.IO.BinaryReader din) [0x00000] in <filename unknown>:0
at ExitGames.Client.Photon.Protocol.deserialize (System.IO.BinaryReader din, Byte type) [0x00000] in <filename unknown>:0
at ExitGames.Client.Photon.Protocol.DeserializeParameterTable (System.IO.BinaryReader binaryReader) [0x00000] in <filename unknown>:0
at ExitGames.Client.Photon.Protocol.DeserializeOperationResponse (System.IO.BinaryReader din) [0x00000] in <filename unknown>:0
at ExitGames.Client.Photon.PeerBase.DeserializeMessageAndCallback (System.Byte[] inBuff) [0x00000] in <filename unknown>:0
at ExitGames.Client.Photon.EnetPeer.DispatchIncomingCommands () [0x00000] in <filename unknown>:0
at ExitGames.Client.Photon.PhotonPeer.DispatchIncomingCommands () [0x00000] in <filename unknown>:0
at ExitGames.Client.Photon.PhotonPeer.Service () [0x00000] in <filename unknown>:0
at Photon.MmoDemo.Client.GameStateStrategies.WorldEntered.OnUpdate (Photon.MmoDemo.Client.Game game) [0x00000] in <filename unknown>:0
at Photon.MmoDemo.Client.Game.Update () [0x00000] in <filename unknown>:0
at MmoEngine.Update () [0x00000] in C:\Users\Amir\\98 MMOTest1\ExitGames-Photon-Server-SDK_v3-0-19-2868-RC8\src-server\Mmo\Photon.MmoDemo.Client.UnityIsland\Assets\Photon\MmoEngine.cs:150
UnityEngine.Debug:Log(Object)
MmoEngine:Update() (at Assets/Photon/MmoEngine.cs:158)

I have played abit with the server code so the error could have been caused by the added code. However I only need to send a smaller array in order to not get this error. for example sending array with the size of 15000 is fine however when sending array with the size of 20000 the error appears so im thinking its at photon's serializing code.

FYI Im using photon server to retrieve data from MongoDB and then passing it to Unity3D. I am trying to figure out where the problem is so that I may decide to use a different method or figure out a workaround.

I am wondering if this is
1) a bug or
2) photon server limitation or
3) lack of understanding on my part ( I am new to software development in general)

Tobias

In general, the limit is 32k array entries, as we use 1 short to carry the length.

I couldn't reproduce the issue here, despite sending 20k doubles.
I tested with the latest RC server and the latest client library.
It seems you use the server and the client library from the Photon-Server-SDK_v3-0-19-2868-RC8? If so, then you should update and try again.

As general advice: If you could, you should avoid such big data amounts in single events/responses. They create a lot of reliable udp packages which all have to arrive in order before they get re-assembled and dispatched. It also uses bandwidth for hauling data that should be preserved for realtime data transfer.

amiractivate

Hi Tobias,

Thanks for the quick reply.

Tried again with RC 9. so far the mentioned errors no longer appear

I am however getting "Received unknown status code: QueueIncomingReliableWarning".

Searching through the forums, I found a post from Kaiserludi saying

The QueueIncomingReliableWarnings mean, that the underlying Photon core client is receiving way more incoming data packets, than you are processing. Normally this indicates, that you are not calling service() often enough

I guess that means ill have to improve on the way I send messages? smaller chunks/ less frequent messages? is that right?

With regards to the data transfer, what I am trying to do is transfering of audio (wav file). Any suggestions on how to go about doing this in a better way (apart from splitting it into smaller pieces)?

dreamora

how often are you sending it? And where are you sending it from?

I assume once and then you don't send anything anymore for the next X to XX seconds as thats what you would need to do when you try to fire around 160kb unless you sit on a 10-100-1000GBit upstream connection for the sender

amiractivate

Hi dreamora,

This is roughly my current setup :
unity3d client (using PhotonNetwork plugin)->RC9 (loadbalancing instance) -> Mongodb (database)

currently im getting this error after sending an "OpCustom" from unity3d and before receiving response from "OnOperationResponse". The content of the OpCustom is a request to get data from the database. The response to this request is an array of float (30k length , about 240k bytes)which is sent via parameterCode.

the sending of the "OpCustom" is triggered only once by a mouse click. This error doesnt appear when Im receiving a smaller array ( 25k length, about 180k bytes).

*im thinking ur assuming this is happening due to multiple requests, but im only sending 1 request.

btw are u the same dreamora from the unity forum/unity answers?

Tobias

You probably can't avoid the QueueIncomingReliableWarning message for this big reply but it's not an error message:

QueueIncomingReliableWarning means that the client queues a lot of reliable commands for you. The queue becomes smaller by dispatching the received commands usually. This is hopefully also described in the reference doc of the client.
In your case, a single, long array is sent and has to be re-assembled. This is only possible once it's received completely. So the client must receive all fragments, before the message can be dispatched.
The warning you get informs you that your in-queue is growing. This is ok in your case but not ok if the server constantly sends more info than the client dispatches. The latter case is why we have that warning.

amiractivate

Thanks Tobias for the clarification