Errors on local network

I have recently started working on networking our Unity game using Fusion and have encountered a strange issue.

I have been finding that when I try to connect to the game using certain computers that are connected to my local network I am seeing these issues.

I have several computers that I have tested with on my local connection (2 desktops and 2 laptops) and only certain combinations have the issues. I have tested both wired and wifi (and combinations of the two) and it makes no difference.

laptop 1 + laptop 2  ->  issues

laptop 1 + desktop 1  ->  issues

laptop 1 + desktop 2  ->  no issues

laptop 2 + desktop 1  ->  issues

laptop 2 + desktop 2  ->  no issues

desktop 1 + desktop 2  ->  no issues

I have also tried connecting laptop 1 to a remote network and joining a game with desktop 1 and found there to be no issues (these same two did have issues on the local network).

The issues that I am seeing are an unusually bad round trip time (0.07 - 0.11) and errors and glitchy behaviour that completely breaks the game when there are more than a certain number of network objects in the scene (around 150).

The behaviour that I see is different depending on which simulation mode I use (delta snapshots / eventual consistency).

Eventual consistency mode:

The network objects on the client will start to move/spawn in a glitchy way, some will be clipping through the ground, some will roll around and ignore the physics constraints that should be there, some will appear and disappear in random places. After a short time of this everything will disappear. No errors show in this case.

Delta snapshot mode:

Everything looks fine until the client starts to spit out errors then completely disconnects from the host after 64 errors:

[Exception] InvalidOperationException: Not Found: [Tick:1905]
Simulation.FindSnapshot() at Fusion/Fusion.Runtime/Simulation/Simulation.cs:1624
Simulation+StateReplicatorDeltaSnapshot.RecvPacket() at Fusion/Fusion.Runtime/Simulation/Simulation.StateReplicatorDeltaSnapshot.cs:165
Simulation+Client.RecvPacket() at Fusion/Fusion.Runtime/Simulation/Simulation.Client.cs:284
Fusion.Simulation.Fusion.Sockets.INetPeerGroupCallbacks.OnNotifyData() at Fusion/Fusion.Runtime/Simulation/Simulation.INetPeerGroupCallbacks.cs:292
NetPeerGroup.HandlePacketNotifyData() at Fusion/Fusion.Sockets/NetPeerGroup.cs:734
NetPeerGroup.HandlePacket() at Fusion/Fusion.Sockets/NetPeerGroup.cs:617
NetPeerGroup.Receive() at Fusion/Fusion.Sockets/NetPeerGroup.cs:564
NetPeerGroup.Update() at Fusion/Fusion.Sockets/NetPeerGroup.cs:143
Simulation.NetworkRecv() at Fusion/Fusion.Runtime/Simulation/Simulation.INetPeerGroupCallbacks.cs:101
Simulation.Update() at Fusion/Fusion.Runtime/Simulation/Simulation.cs:875
NetworkRunner.UpdateInternal() at Fusion/Fusion.Runtime/Runner/NetworkRunner.cs:1435
NetworkRunnerUpdater.InvokeUpdate() at Fusion/Fusion.Runtime/Runner/NetworkRunnerUpdater.cs:75

I have tried simulating a bad network connection with a remote player (using the network condition options in the fusion debug dll) and even when simulating a much worse rtt (0.13 - 0.16) couldn't get any errors to happen in delta snapshot mode. In eventual consistency mode I did eventually get an error but only when there was a very large number of network objects (around 700):

[Exception] AssertException: Exception of type 'Fusion.AssertException' was thrown.
Assert.Fail() at <118f5685e46d4d828df8043f76904869>:0
SimulationConnectionObjectData.SetSentTick() at <e9d9a7bd03474f788eb2d43d36c93184>:0
Simulation+StateReplicatorEventualConsistencyBase.OnPacketLost() at <e9d9a7bd03474f788eb2d43d36c93184>:0
Simulation.OnEnvelopeLost() at <e9d9a7bd03474f788eb2d43d36c93184>:0
Fusion.Simulation.Fusion.Sockets.INetPeerGroupCallbacks.OnNotifyLost() at <e9d9a7bd03474f788eb2d43d36c93184>:0
NetPeerGroup.HandlePacketAcks() at <3bb1704e165c465fac54c58a29c3078b>:0
NetPeerGroup.HandlePacketNotifyData_Part2() at <3bb1704e165c465fac54c58a29c3078b>:0
NetPeerGroup.HandlePacketNotifyData() at <3bb1704e165c465fac54c58a29c3078b>:0
NetPeerGroup.HandlePacket() at <3bb1704e165c465fac54c58a29c3078b>:0
NetPeerGroup.Receive() at <3bb1704e165c465fac54c58a29c3078b>:0
NetPeerGroup.Update() at <3bb1704e165c465fac54c58a29c3078b>:0
Simulation.NetworkRecv() at <e9d9a7bd03474f788eb2d43d36c93184>:0
Simulation.Update() at <e9d9a7bd03474f788eb2d43d36c93184>:0
NetworkRunner.UpdateInternal() at <e9d9a7bd03474f788eb2d43d36c93184>:0
NetworkRunnerUpdater.InvokeUpdate() at <e9d9a7bd03474f788eb2d43d36c93184>:0

So in conclusion there are two issues:

1) For some combinations of computers on my local network I am getting an unplayably bad rtt. These same computers are fine when connected remotely. Some combinations of computers have no issues when connected on the local network.

2) In these cases I get errors/glichyness/failures rather than the normal behaviour that I see when I simulate a bad connection. When I was simulating a really bad connection I could get to 1000+ network objects with no errors (in delta snapshot mode) but in these special cases I can only get to around 150-200 objects before everything breaks.

I am on the latest version of fusion (0.13.0) and I am on unity version 2020.3.12.