Problem updating position of grabbable GameObject using Photon2 and Unity3D

Options
I'm trying to add Photon2 support to a simple Oculus starter scene (cubes on a table). I was able to run the Avatar synchronization code described here https://doc.photonengine.com/en-us/pun/v2/demos-and-tutorials/oculusavatarsdk on top of my basic Photon2 setup.

This is the current sequence:
  • Player 1 ( P1 - the one using Quest) creates the room and joins it

  • Player 2 (P2 - the one using Unity Player) joins the room

  • Player 1 grabs a cube and moves it around
P2 sees the cube moving normally if P1 just drags it horizontally, but if he tries to lift it from the table the cube seems acting crazy from P2 point of view.

It looks like the cube fights with gravity but I'm not totally sure what is going on. You can see an example in this video where there is an alternation between horizontal drag and lifting from the table: https://gofile.io/?c=A9PN41

If I disable gravity on the cube it seems working fine (although the interpolation a bit slow compared to how P1 is moving the object): https://gofile.io/?c=jM4qEE

The point is that I cannot have cubes flying around the scene :)

Did anybody incur in such a problem?