Photon Server SDK stutter / lag issues when run on one machine, but not the other

owenhindley
edited January 2022 in Photon Server

Hi! I'm developing a local multiplayer VR experience based on Photon for 5 Quest headsets + one desktop Operator client (A development on from this project http://huldufugl.is/kassinn).

I've recently come across a major issue when running the server (v5.0.12) on our 'show machine' versus my 'development machine'.

I've made a minimal test setup consisting of two Unity editors running on each machine, that connect to a local server via IP, and spawn a NetworkPrefab with a Voice Recorder playing back a piece of VO, and a Speaker for its remote counterpart.

When the server is run on my development machine, audio passes correctly and stutter-free between show machine and development machine. However if the server is run on the show machine, I encounter severe and very regular stuttering lik-e-si-len-ce-is-in-ser-ted-in-bet-wee-nfr-a-mm-es.

Checklist:

  • Windows Defender is disabled on both machines
  • Both machines connect via a standalone router via Wifi
  • This is extremely repeatable
  • I've copied the server SDK folder from the working dev machine to the non-working show machine, to ensure that (other than IP settings in the various .config files) all settings are comparable
  • Both machines are good spec (show machine is a laptop, but is actually higher spec than development machine - i9 vs i7)
  • Logging has been disabled on both machines via Photon Control
  • in this test scene, no other data is being sent other than voice
  • Similar CPU load on both machines
  • Behaviour is the same in 'DemoVoiceMinimal-Scene'

Observations

  • When stuttering occurs, the Photon Speaker lag counter (above the frequency graph) oscillates wildly between 200/300 and a low number, which I swear sometimes goes negative. When the server is run on the working machine, this number remains stable around 250.
  • This also seems to affect normal Photon messages like RPC calls and serialized data. When the full show is running, data and voice is subject to many seconds of lag, and broken up as above.

I've included logs from both the development server and show machine.

Any advice you may have would be very much appreciated!

Comments

  • Apologies, just read the bug report guidelines, here's the config for both machines (they're identical as mentioned above, save for the IP addresses) :


  • Here's a log file from the development machine. The 'working' server is at 192.168.0.101, and the 'stuttering' server is at 192.168.0.100.

    And here's an example of negative lag :


  • chvetsov
    chvetsov mod
    edited January 2022

    hi, @owenhindley

    first what I see in your logs is that debug logging is active - around 250 messages per second. this is usually the reason for stuttering.

    go to LoadBalancing/GameServer/bing/log4net.config and switch off debug logging

    do the same for master server

      <root>
        <level value="INFO" />
        <appender-ref ref="LogFileAppender" />
        <appender-ref ref="ConsoleAppender" />
      </root>
    

    best,

    ilya

  • Hi Ilya,

    Okay wow - that seems to have done the trick!

    FWIW - I'm 99% sure any logging settings I would have enabled via Photon Control, and as part of my attempts to fix this, I definitely tried all the available 'disable logging' / 'remove counters' options within Photon Control.

    Is it possible that Photon Control is enabling some settings like the one above, but not disabling them correctly when requested by the user?

    Either way, thanks for the input!

    Owen

  • Photon Control does not update those configs as far as I know

    best,

    ilya

  • Hi Ilya,

    So that value (<level value="INFO"/>) was set to 'DEBUG' in my GameServer log4net.config - and setting it back to 'INFO' did solve the problem.

    But I definitely didn't manually edit that file before - if Photon Control doesn't change that value, is there some other way by which that value would have been set to 'DEBUG' ? i.e. from a Unity client, is that possible?

    I'd like to avoid this happening again, so just trying to understand how it got that way to begin with.

    Owen

  • that value is by default DEBUG.

    if you are building LoadBalancing solution you have to update it right in solution. Find LoadBalancing project and log4net.config in it

    if you are not building than changing in LoadBalancing/GameServer/bin folder should be enough

    best,

    ilya