Objects respawn at initial location as soon as new player enters or leaves the room

Options
Hi all, I am new to Photon and have an issue right away:
my gameObjects (other the the actual players) respawn at their initial position as soon as a new player enters the room.
Let's call my object Ball. I tried this:

I put a prefab of Ball in resources, add photonview component and simply place the Ball in the Hierarchy of Unity.
On the BallScript, included Photon.Pun and in the Update start with this line:

if (!PhotonNetwork.IsMasterClient) return;
// some movement for testing sake here

The idea is to have the ball to be owned by the Scene and therefore be controlled by the MasterClient only (am I right here?)

The problem rises when new players enter the room OR leave the room (that is: I open or close several standalone Builds in Windows). All clients sync fine and show the moving ball on the same position! But as soon as I add or remove a client, it seems to restart the Scene! Iin the remaining clients the ball respawns at the original location.

Any help would be appreciated!