Excessive client memory allocation

Jormungandr

I'm using Photon 2.4.7 (soon to upgrade to 3, we hope) with Unity 3.4.2.

I see a whole lot of memory allocated every time I call PhotonPeer.Service(), on the order of a few dozens of KB. This is slightly problematic: Unity's less-than-stellar garbage collection stops the whole game loop for noticeable periods of time every time the garbage collector runs, and continuous memory allocation like this increases the frequency with which those stutters occur. These are not little stutters, either - they're close to half a second of total unresponsiveness, which in a realtime game is very, very noticeable.

You can watch the entire process in Unity's built-in profiler; deep profiling pretty clearly indicates the memory is being allocated by a bunch of array operations in the guts of EnetPeer.serializeToBuffer() -> NCommand.serialize() -> SupportClass.NumberToByteArray().

So, my questions to you, in order:
1. Is there anything I can do to ameliorate this? Preferably things other than "call Service() less often" or "send less data".
2. Has this issue been addressed in more recent versions of the Photon client? If the answer to this is "no", I'd like to file a bug.

dreamora

Deep profiling is never an option if you want to check such a thing, its overhead is massive enough to generate issues where none existed before actually.
If you are on osx you better use instruments & shark vs deep profiling.

also that an allocation happens is unavoidable, you request to get data sent and serialized so Whatever -> byte[] upon send and byte[] -> whatever has to take place which means that you have the temporal byte[], no matter if you want or not.
If you want to overcome it, stop serializing stuff around and keep all your data in byte[] containers all the time.

And its not unitys GC, its Mono 2.6 GC actually and its normally good enough when not forcefully pushed over the edge. If you get dozens of kb per call that it has to serialize then I would check if you aren't forcefully trashing the pipeline as that sounds like a lot of data.

1. Well call it less often would be the way to go especially if you are at a rate is >= 50 (20ms latency is pretty much never gonna happen so there is no reason to fire it at this rate)
The other solution naturally is better interest management. stop sending data to clients that don't even need it, to reduce the general amount of traffic floating around. So a 'room wide' event might not always be the smartest idea.

2. Serialization needs temporal byte container to send, there is no bug nor a way to avoid it. But I've not seen any GC spikings with Photon 3 & Unity 3.4 so far. (I refuse using 3.4.2 though due to the problems it introduced, I'm on 3.4.1)

Jormungandr

Deep profiling is quite accurate in this case; the overhead you speak of is per-function CPU, not memory allocation.

Somewhere along the line, yes, byte arrays must be allocated; that doesn't stop reasonable steps from being taken to minimize that allocation (perhaps reusing relevant buffers rather than trusting the GC blindly). Believe me, I've done this enough in my own code. Since I don't have access to the photon client source, I can't speculate as to the exact cause of these problems - but they are problems, and they can be reduced even if not eliminated.

Yes, it's the Mono 2.6 GC. Which is going to remain the one used by Unity for the foreseeable future, so my points stand. I will certainly investigate what I'm sending, but past measurements of our data rate suggest this allocation is disproportionately high.

These GC spikes are hardly unique to recent versions of Unity. They've been present since the 2.x days, at least, and the symptoms that exaggerate them have remained the same since then as well. Try doing anything with OnGUI, for example, (and this is the reason we wrote our own UI library) to see what I mean. Or, really, anything that allocates more than a few bytes of heap memory every frame.

dreamora

Actually you have access to the client code in Photon 2 if you talk about the dotnet dll thing.
its in the server sdk, the top project if I remember right.
For unity its the same just that you have to hand modify a few things (replacing system log output with unityengine.debug.log and other obvious things)

In Photon 3 you are right, there the project that builds this dll is not available at least at the time.

As for 'caching the buffer': don't know out of head if thats technical even possible.

And deep profiling is a major problem in messing up things, you can believe me or not but the state the deep profiler is in makes it a 'no go' if you want correct data.
I normally use shark + instruments on osx for this very reason because they give me accurate data not 'good luck, heavily runtime behavior modified' deep profiling informations which tend to be between slightly wrong and completely unrealistic. the presence of the deep profiler itself for example already generates a fair amount of trash for all the lookups to store the 'time slices'

And yes the GC spikes were always around though since Unity 3 its far less problem than on mono 1.3.x on which Unity 2.x was sitting.
And OnGUI is nothing that has to do with this here, we talk about the networking and things relevant to it. Unity has its problems yeah but it also has its board to talk about them
Unity OnGUI is a problem cause its a hog and badly documented on how to use it right and do it right. cause GUIX already in unity 2.1 has proofen that you can raise the performance by a factor of 10+ with unity gui without a problem.

Jormungandr

As far as I know, deep profiling in Unity introduces the following issues:
- Additional application memory footprint for the instrumentation and sample storage
- Additional per-function CPU overhead

None of which affects the values displayed for per-frame allocation, which is what I'm discussing. If there are other, specific, issues introduced by deep profiling that you are aware of, please share, but "it's full of crap" isn't useful.

In any case, the issue remains regardless of the profiling method used (statistical or fully instrumented).

I mention OnGUI only as an example of another allocation-happy system that introduces GC spikes.

I would also still like to see what one of the actual Photon devs thinks of the matter.

Tobias

That's quite a coincidence.
We dived into performance related topics in the the de/serialization recently and made good progress there (that's the reason why I didn't show up in the forum). Our changes are geared towards the scenario "Unity as headless server for in-game logic" and we focused on CPU performance. Still, as side effect, we already improved memory impact as well.

The improvements are Photon 3 only and not released yet. I could update you with a preview.

What is your target platform and usecase?

Jormungandr

Our server is Photon + custom C# game logic, using Unity as the client. Target platforms are web and desktop.

The game uses a fully authoritative server, sending data from server to client through our own object view / delta compression scheme built atop Photon. The client currently calls Service() 20 times per second.

Since I started this topic we've successfully finished migrating to Photon 3, so I would be quite interested in trying that preview.