OVRInput.Get stops working when instantiating local VR player

Options
I'm having a weird issue, and while I don't think it's PUN's fault I'm hoping someone here may have run into the same issue and found a solution.

When using PUN, the right thing to do (as I understand it) is to have the player be a prefab, and instantiate it on connect so the player is shown remotely too.

Now, because the connection sometimes takes a little bit and I don't want to have just a black screen while it's going on, I have put an OVRCameraRig in my scene, so the camera is active and the user can see stuff (and move their head) as the game is connecting. All good so far. When we're connected, I destroy that camera rig and instantiate a new OVRCameraRig prefab as explained in the "Oculus Avatar" document linked earlier.

Here's the weird thing: even though the camera works just fine and the user can look around (and see their hand), reading the status of the controller buttons does not work anymore: OVRInput.Get always returns false when I ask whether a button is pressed. I'm using this code to check the button:
bool pressed =  (OVRInput.Get(OVRInput.Button.PrimaryTouchpad));
If I remove the initial OVRCameraRig then things work again (including detecting button presses), but we have the black screen for a while during connection.

Has someone else run into this, and found a way to switch from one OVRCameraRig to the other that works? I'm trying to avoid the black screen, or loading a whole new scene for now.