Getting huge message spikes, are there detailed message analytics?

Rights

Hello,
we're working on a realtime multiplayer mobile game using Photon 2. We're currently facing a problem with huge message numbers in certain areas, and are unable to trace the problems back to a source due to lack of analytics data. We have a solid knowledge of what goes through to the server from the client, but it seems somewhere in this flow something is causing massive message numbers and we don't really know what.

Is there any way to get more out of the analytics collected by Photon? We can see most areas running fine with less than 500 messages, but a select few (USA is one) are getting 60k+ message spikes every now and then with ~6 players in a room at once.

We've been optimising our network code for weeks now (stuffing everything possible into byte arrays, local calculation instead of networked etc.), and every time we think we reached a solution a new spike appears after a while.

Does Photon collect logs of sent/received messages on the server side that are available for developers to view? If not, how would we go about creating a log system to trace sent/received message information on the client? I don't need straight up code, I just need a red line to follow.

Google isn't very great when it comes to photon services...

Thank you in advance!

JohnTube

Hi @Rights,

Thank you for choosing Photon!

60k+ message spikes

wow! that's too many. how do you easily miss that?

Did you consider that there are some hacked clients that got hold of your AppId?

Does Photon collect logs of sent/received messages on the server side that are available for developers to view?

No. If we notice that one application is doing something wrong and may affect other applications on the public cloud, we get in touch with the developer or take it to a separate cluster temporarily where we could investigate if necessary. But this is not something we do often.

PUN has Stats GUI Component ICYMI.

how would we go about creating a log system to trace sent/received message information on the client?

First I would make sure we understand how messages are counted (number of players, send rate, serialization rate). See our docs and the FAQ or some forum messages (here or on Unity's forum). Or ask us directly.
Then I would start by increasing PUN logging levels (that may not be very useful).
I would log all Instantiate, RPC, RaiseEvent, SerProperties methods calls.
I would add also some log calls in PUN's code here and there where we call LoadBalancingPeer.SendOutgoingCommands(), see PhotonHandler.

Rights

JohnTube said:

Hi @Rights,

Thank you for choosing Photon!

60k+ message spikes

wow! that's too many. how do you easily miss that?

Did you consider that there are some hacked clients that got hold of your AppId?

Does Photon collect logs of sent/received messages on the server side that are available for developers to view?

No. If we notice that one application is doing something wrong and may affect other applications on the public cloud, we get in touch with the developer or take it to a separate cluster temporarily where we could investigate if necessary. But this is not something we do often.

PUN has Stats GUI Component ICYMI.

how would we go about creating a log system to trace sent/received message information on the client?

First I would make sure we understand how messages are counted (number of players, send rate, serialization rate). See our docs and the FAQ or some forum messages (here or on Unity's forum). Or ask us directly.
Then I would start by increasing PUN logging levels (that may not be very useful).
I would log all Instantiate, RPC, RaiseEvent, SerProperties methods calls.
I would add also some log calls in PUN's code here and there where we call LoadBalancingPeer.SendOutgoingCommands(), see PhotonHandler.

Hey!
Thanks for the help!

I actually wrote this a bit after seeing some of the analytics data, but as I can't access it myself (no team option for multiple accounts in analytics panel?) the amount of messages I said was wrong. Luckily it's 'only' in the 1000 message range, so not 60k. It was the data amount that I was looking at before moving back to my seat to write my message. Nevertheless it's still a spike we need to get rid off.

We didn't really 'miss' the message spikes, they're currently appearing in our beta since last week, and are a surprise to us, also. Hence they're also currently in our top priority list of fixes.

We've noticed some versions on Android possibly being hacked already, which isn't that surprising to be honest... We're also working on that.

Right after posting my message, we found the Stats GUI component and started using it. Already noticed some possible places for improvement, but none of these should create the spikes we see here, unless there's some super illegal going on with hacked clients. Nevertheless this will help us, thanks!

Really enjoying what Photon has to offer, keep up the good work!