Lag using ExecutionFiber

Hi,

We've been experiencing some lag issues every now and then, mostly when we reach ~100 CCU. The game is simply ticking every 100 ms and publishing an event. The server resources aren't a problem as we perfmon'ed the whole process without noticing anything. We've also profiled our code and the logic inside the tick does not appear to be the problem but the time interval between ticks are greater than 100 ms when lag is observed. It might be important to add that we perform database transactions upon initialization and finalization of each game, where we synchronously wait for the transaction to finish. We assumed that this wouldn't affect other games, but are unsure, since we have not been able to track down the issue.

Could there be something specific in photon that could cause such a lag ?

Regards.

Comments

  • Tobias
    Tobias admin
    edited January 2017
    Sorry you didn't get a reply. The server guys didn't find this, I guess.
    Did you solve the riddle or still need help?

    I guess this answers it: http://forum.photonengine.com/discussion/8603
  • kennethharder
    edited January 2017
    It did not get answered. The other post got "answered" a few days before this one.

    This solved the issue:
    try
    {
        // Change the number of threads.
        int minWorkerCount, minIoCount;
        int maxWorkerCount, maxIoCount;
        ThreadPool.GetMinThreads(out minWorkerCount, out minIoCount);
        ThreadPool.GetMaxThreads(out maxWorkerCount, out maxIoCount);
    
        minWorkerCount *= 8;
        minIoCount *= 8;
    
        maxWorkerCount = Math.Max(minWorkerCount, maxWorkerCount);
        maxIoCount = Math.Max(minIoCount, maxIoCount);
    
        ThreadPool.SetMaxThreads(maxWorkerCount, maxIoCount);
        ThreadPool.SetMinThreads(minWorkerCount, minIoCount);
    }
    catch (Exception e)
    {
        Logger.LogException(e);
    }
    
  • what is going one with your game is that you block to many worker threads. that is why rest of threads can not execute all task fast enough.

    in general solution you found will NOT help you when you will have more games, because amount of threads is finite.

    so, i would recommend you to change database update logic, so that it performed asynchronously and at the end enqueues action to rooms fiber.

    Probably usage of RegisterWaitForSingleObject will help you to resolve this issue
  • The profiler showed that the number of worker threads was low. Less than a 10. So that's not the issue.
    We experience lag because the number of worker threads gets automatically adjusted by .NET. As the fiber tasks are very short in most cases and the .NET threads are getting reused for multiple fibers, everything goes fine until we queue something expensive. When that happen the task is queued on one of the worker threads and is stalling every tick that gets queued on top of it. This is because the .NET threadpool isn't very reactive when it comes to adjust the number of threads.
    This is confirmed by several observations.
    The fact nobody isn't lagging during the first 10 mins because the threadpool has allocated many worker threads on startup that haven't yet been put to sleep. Thereby granting a larger choice of threads when queuing new tasks.
    The fact that the lag appears with only 4 players.
    The fact that so far the fix has scaled up to 500+ players.
    The fact that we only block one thread when we experience the lag with 4 players.