State replication issues with VR.

Muzuka

Hi I'm a Bolt newbie but have been working on this problem for a while and need some pointers.

I'm working on an online VR game in Unity and trying to implement syncing of the hands and head. I've gone through a few ideas but for some reason, the server isn't replicating the client's movement. I would like some help with figuring out what I'm doing wrong with the state replication.

I am using VRTK 3 so each player has multiple input avatars that VRTK chooses from to load. I have a system working where I let the client load the avatar and once it finishes it tells the server which one was loaded and the server manually activates the hands and head and attaches the avatar models that should be shown to the other clients.

I have a Bolt state asset with transforms for the player position, head, and two hands. The client uses setTransforms when the input avatar is loaded and the server does the same when it loads its avatar. VRTK gives you references to the loaded hands and head objects so I know the client and server are setting the same transforms for replicating. When I run it the server has the avatar loaded but the components aren't following their counterparts on the client at all. The server avatar is all stuck at (0,0,0). I've tested with multiple clients and the avatar is visible but it's stuck at the same position.


This is how my state is set up.

using System.Linq;
using VRTK;
using static VRTK.VRTK_SDKManager;
using UnityEngine;
using Bolt;

public class VRPlayerBehaviour : EntityEventListener<IVRPlayer>
{
    private VRTK_SDKManager _vrManager;

    private bool _isSetupLoaded = false;

    public override void Attached()
    {
        BoltConsole.Write(entity.name + " attached.", Color.green);
        BoltConsole.Write("Controller = " + entity.IsControlled, Color.green);
        BoltConsole.Write("Owner = " + entity.IsOwner, Color.green);

        _vrManager = GetComponent<VRTK_SDKManager>();
        _vrManager.LoadedSetupChanged += SetupVR;

        if (BoltNetwork.IsClient && entity.IsControlled)
        {
            _vrManager.TryLoadSDKSetupFromList();
        }
    }

    private void SetupVR(VRTK_SDKManager sender, LoadedSetupChangeEventArgs e)
    {
        int setupIndex = sender.setups.ToList().FindIndex(x => { return sender.loadedSetup.name == x.name; });
        
        state.SetTransforms(state.PlayerTransform, sender.loadedSetup.actualBoundaries.transform);
        state.SetTransforms(state.HeadTransform, sender.loadedSetup.actualHeadset.transform);
        state.SetTransforms(state.LeftHandTransform, sender.loadedSetup.actualLeftController.transform);
        state.SetTransforms(state.RightHandTransform, sender.loadedSetup.actualRightController.transform);

        sender.loadedSetup.GetComponentsInChildren<VRAvatarSwitcher>().ToList().ForEach(x => { x.SetModel(false); });

        BoltConsole.Write("VR setup has changed to " + setupIndex, Color.green);
        _isSetupLoaded = true;

        VRSetupNotify vrSetupEvent = VRSetupNotify.Create(entity);
        vrSetupEvent.SetupID = setupIndex;
        vrSetupEvent.Send();
    }

    public override void OnEvent(VRSetupNotify evnt)
    {
        if (BoltNetwork.IsServer || !entity.IsControlled)
        {
            // load setup manually
            VRTK_SDKSetup clientSetup = _vrManager.setups[evnt.SetupID];

            _vrManager.enabled = false;
            clientSetup.gameObject.SetActive(true);
            clientSetup.actualHeadset.SetActive(true);
            clientSetup.actualLeftController.SetActive(true);
            clientSetup.actualRightController.SetActive(true);

            clientSetup.GetComponentsInChildren<VRAvatarSwitcher>().ToList().ForEach(x => { x.SetModel(true); });

            state.SetupID = evnt.SetupID;
            state.SetTransforms(state.PlayerTransform, clientSetup.actualBoundaries.transform);

            state.SetTransforms(state.HeadTransform, clientSetup.actualHeadset.transform);
            state.SetTransforms(state.LeftHandTransform, clientSetup.actualLeftController.transform);
            state.SetTransforms(state.RightHandTransform, clientSetup.actualRightController.transform);

            _isSetupLoaded = true;

            BoltConsole.Write("Loaded vr setup from client.", Color.green); 
        }
    }
}

Here is the EntityListener script.

The client calls SetupVR when the setup is loaded. It then sends an event to the server. The server calls OnEvent when it receives the event.

Is there anything obvious I missed or debug steps I could take? I'm sorta scratching my head cause it's set up the same as I did in the basic tutorial and the only thing I changed was the state transforms replication mode to "everyone except controller."

I have a bonus question too. Another idea I tried was to have a separate avatar from the input avatars. I wanted to have the input avatars transform positions be transferred to the server's avatar transforms somehow. I'm not sure how viable this is though. Would I have to use commands?

stanchion

Take a look at this VRTK4 sample, it is pretty much the same idea for VRTK3 or any other VR SDK.
https://trello.com/c/K08PzVcy/4-vr-bolt

I'm not sure what you mean by "separate avatar"

Muzuka

Thanks for the link! That looks really helpful. The main difference with VRTK 4 and 3 is 4 has one input rig/avatar while 3 has multiple that VRTK chooses at startup. The server doesn't know which one gets loaded right away so I can't set the transform in attached.

I'm guessing calling SetTransforms in OnEvent might be my main problem.

What I mean by a separate avatar is putting the visible avatar models into a separate hierarchy from the input rig/avatar but still under the bolt entity. What the bonus question was asking was if it's possible for the client and server to set different transforms to a transform in a state and have them track each other. The controlling client moves the transform it set and on the server, the second transform could follow it.

I realized though the simpler solution would be to have the server and client track the visible avatar and have the visible avatar controlled by the client. I'll try that and see.

Muzuka

The example answered my bonus question actually. Thank you! It passes two transforms into one state transform and it works. The example project works fine for me too.

For some reason, SetTransforms still doesn't work for me in my project. I implemented the block example in the same project and that works fine. In my VR scene, it doesn't work though. I tried adding a callback and it's not getting called when I move a tracked transform so maybe the connection to the states not being made?

Muzuka

I've figured out the issue. I believe it was an ownership issue. The server owns the player object so it determined the values of the transform. Moving the VR controls on the controlling client's side wouldn't replicate the values so they would always follow the server which sets them to the default (0, 0, 0) position.

I need to transfer the VR's input to the server either with commands or events. I'll try making the states changeable by the controller and use events to update the server. If this doesn't work I'll use commands.