Restarting Server fixes issues

Options
I have created a FPS shooter game that has been handling 20 players just fine. I am using Photon Server v5 and also have the Photon Server Plugin 5 beta downloaded and I use them both depending if there are issues.
I have a few problems relating to both the server and the plugin.
But here is some context first.

The game has been built with the help of Simple https://doc.photonengine.com/zh-cn/pun/current/simple/simple-tutorial.
Simple is used to sync the animation, position of characters, also the position of what the players are looking at and the position of where they are aiming. I am using PunRPCs to send information when to begin shooting and when to stop shooting. PunRPCs are also used for weapon pickups, weapon changes, and weapon drops.

Everything is working fine and smoothly however after a certain amount of time (a day, sometimes a few hours, sometimes a couple days) the server stops running properly. Room instances stops sending messages to the clients, or stops setting room properties. Games will not start because of it as they require some of these room properties to start. Amazingly games that do start, lag is introduced with a massive backlog of messages it seems. For example, a game that used to have only 200ms of lag due to the server location, will have about 15 seconds of lag. It seems to get progressively worse the longer the server is running. The only solution is to restart the server. Then it works flawlessly.

So in my head, data is accumulating on the server causing congestion. We had problems with even 2 people in a room.

I have run into a couple issues with the plugin sdk and the server sdk.
1. The follow error is on repeat. System.DllNotFoundException: Unable to load DLL 'PhotonCryptoPlugin': The specified module could not be found. (Exception from HRESULT: 0x8007007E).
Due to this error I tried to use the full server instead.
2. When I started using the server sdk, I recieve these messages on repeat when a player joins. (I suspect these messages are from how Simple works, because when I disabled the NetObject the messages disappeared... however they don't appear whilst using just the plugin)
2021-06-04 23:37:25,694 [29] DEBUG Photon.Hive.Plugin.RaiseEventCallInfo - Continue.
2021-06-04 23:37:25,694 [29] DEBUG Photon.Hive.HiveGame - Broadcasting: Recipients selected basing receiver group: Others
2021-06-04 23:37:25,731 [24] DEBUG Photon.Hive.HivePeer - OnOperationRequest. Code=253
2021-06-04 23:37:25,731 [22] DEBUG Photon.Hive.HiveHostGame - Broadcasting: Got RaiseEvent from 1/'77f9c9da-5910-477a-b99f-35022ce1870a', targetActors=-1, Group=0, ReceiverGroup=0
I have not enabled the debugging, I have looked through all the config files and debugging of this kind is not present and disabled. The config files have not been touched and they are the same for both the plugin and the server except only the server shows these messages.

If I compare both the plugin and the server; the plugin seems to have far less lag introduced. When I first implemented the server with the plugin that I built that I had used in the plugin SDK, there was lag introduced instantly of about 4 seconds. I reverted straight away back to the plugin SDK. But next time I tried the server it worked fine. I want to disable the logging of these messages to see if it fixes these problems but do not know how when the config files already say disabled. Although, I am not sure this is the foundational issue.

I have a feeling that maybe there is some memory problems or leakage which is why I need to restart the server to make things work. There hasn't been more than 20 players on the master server because we haven't openly released the game yet are still having issues.

Any pointers in the right direction would be helpful. I can't exactly restart the server every day to make things stable.

Sorry for the long message but thanks in advance!

Comments

  • JohnTube
    JohnTube ✭✭✭✭✭
    Options
    Hi @iampoole,

    Thank you for choosing Photon!

    I think in your case you need to use a plugin SDK as it makes more sense since you are developing a custom plugin and not modifying server application. See here.
    Please try to migrate to Photon Server Plugins v5 RC1.
    There is a chance you are using Photon Server v5 RC1 and switching to Photon Server Plugins v5 BETA.

    About the lag, I think this could be due to log spam combined with usage of Windows Defender or other AntiVirus/MalwareLiveScanner, etc. Maybe server has limited resources (I/O, CPU, Memory)?
    See the warning here about logging.

    Let's try answering your questions:

    1. You can ignore.
    2. Let's change default logging level (for <root> in "deploy\LoadBalancing\GameServer\bin\log4net.config" and "deploy\LoadBalancing\Master\bin\log4net.config") from DEBUG to INFO or WARN or ERROR.
  • iampoole
    Options
    Thank you very much. Answered my questions quickly and clearly. :)