Can't get movement of player rigid body smooth - help !

Options
I posted up a similar thread a week ago or so, but since that time I went back and looked at everything I could possibly think of and attempted to experiment with all the possible settings I could.

I'm making a Sim where people can fly around quadcopters together, but the remote players always look jerky. Here's a video where I've followed a friend flying as close as I can to show the issue.

https://youtu.be/Sn7yOpsZYV8

You can see that my version of the object is getting a positional update and then jumping to that position, making everything seem very jerky and weird.

In this test I was just syncing position and rotation using the PhotonTransformView (as well as sending some data via OnPhotonSerializeView. PhotonTransformView seems the most basic of the observable components, but seems to be the smoothest I can achieve.

I have used PhotonTransformViewClassic (in various ways, but the strongest recommendation seemed to be with sync on position/rotation using lerp with a value of 5 and teleport if >3. I've also used it (as well as PhotonTransfromView) in conjunction with PhotoRigidBodyView (sync on velocity/angular velocity). Nothing seems to make the movement any better - in fact if anything slightly worse.

I've made sure that nothing in my copy of the remote object isn't doing anything to the object - so it's essentially left to fly through space. It is a rigidbody, I've had better results by turning the local gravity for it off, and experimented with having iskinetic on/off.

I must be missing something crucial - why is lerp not smoothing out the movement, why is the PhotonRigidbody view not keep it going in roughly the right velocity... I'm so confused about why I can't get this.

Can anyone help ?