Sync SceneView with instantitated view

Peaj

I would like to use a scene player object and sync it with avatar prefabs on the other clients.
My first approach was this:

• Add all neccessary views to the player scene object and the avatar prefab
• AllocateViewID for the player
• Instantiate the Avatar via RPC and assign the allocated view id

This does not work though as the player gets a viewID on scene load and it cannot be changed afterwards,
which would lead to all players having the same viewID

Is there a better approach?
Or is it possible to prevent the players view from getting a viewID on scene load or change it afterwards?

Thanks,
Pascal