Spontaneously movings when there are multiple photon view objects

Options
I am making game like jenga blocks that players can take ownership and drag blocks when their turn.

But blocks sliding itself and fall down after i added transform view and rigidbody view on each blocks. How can i fix this problem?