Oculus Quest 2 with hand tracking, grabbable objects not synced.

Matias
2020-11-24 14:45:39

Hi guys, I'm working in VR with the Oculus Quest 2 using Unity 2019.4.14f1 LTS version + PUN 2.

I managed to make hand tracking work and send the hands position and rotation through the network by using this plugin and the help of the developer. The hand's movement works really well without any lag or major latency problems.

Now... I'm having problems related to the synchronization of grabbable objects.

Let's say player1 and player2 are sharing a room. In front of them, there's a cube. If player1 grabs the cube and moves It with his hand, he will see the cube moving around without any problem, but for player2 the cube will instantly disappear as soon as player1 pinch the cube and move it. The cube will be visible for player2 again, only if player1 releases the object, which will suddenly appear in the position where player1 stopped grabbing the cube.

Everything else is working really well, the hand tracking, the avatars, the objects, voice, chat, etc.
I'm using modifications of the OVR grabber and OVR grabbable Oculus SDK scripts.

This is how the components set up for the grabbable object (a cube).
https://ibb.co/drJtF8P

And this is the components set up for the hand prefab I'm using.
https://ibb.co/SPS16wJ

Do you have any clue about this?

Comments

Matias
2020-11-24 14:47:48

This is my custom PunOVRGrabber.cs script.

using Photon.Pun;
using System;
using System.Collections;
using System.Collections.Generic;
using UnityEngine;

namespace Networking.Pun2
{
[RequireComponent (typeof (PhotonView))]
public class PunOVRGrabber : OVRGrabber
{
PhotonView pv;

    //HandTracking related  
    PunOVRHand trackingHand;  
    private float pinchThreshold = 0.7f;

    protected override void Awake()  
    {  
        base.Awake();  
        pv = GetComponent<PhotonView>();  
        if(GetComponent<PunOVRHand>() != null)  
        {  
            trackingHand = GetComponent<PunOVRHand>();  
        }  
    }

    //Basically, if photonview is mine, update anchors from Grabber  
    public override void Update()  
    {  
        if (pv.IsMine)  
        {  
            base.Update();  
            //If hand tracking component detected, check pinch for grabbing mechanism.  
            if(trackingHand != null)  
            {  
                CheckPinch();  
            }  
        }  
    }

    //If the pinch strenght is bigger than the threshold, call GrabBegin(), if smaller, call GrabEnd().  
    private void CheckPinch()  
    {  
        float pinchStrenght = trackingHand.GetFingerPinchStrength(PunOVRHand.HandFinger.Index);

        if(!m_grabbedObj && pinchStrenght > pinchThreshold && m_grabCandidates.Count > 0)  
        {  
            GrabBegin();  
            Debug.Log("Grab begin");  
        }  
        else if (m_grabbedObj && (pinchStrenght < pinchThreshold))  
        {  
            GrabEnd();  
        }  
    }  
}  

}

Matias
2020-11-24 14:48:50

And this is my custom OvrGrabbable.cs script:

using UnityEngine;
using UnityEngine.Events;
using Photon.Pun;

//
//Custom grabbable script which checks if the grabber "is mine" to actually grab
//

namespace Networking.Pun2
{
[RequireComponent(typeof(PhotonView))]
public class PunOVRGrabbable : OVRGrabbable
{
public UnityEvent onGrab;
public UnityEvent onRelease;
[SerializeField] bool hideHandOnGrab;
PhotonView pv;
Rigidbody rb;

    protected override void Start()  
    {  
        base.Start();  
        pv = GetComponent<PhotonView>();  
        rb = gameObject.GetComponent<Rigidbody>();  
    }

    override public void GrabBegin(OVRGrabber hand, Collider grabPoint)  
    {  
        m_grabbedBy = hand;

        pv.TransferOwnership(PhotonNetwork.LocalPlayer);

        if (pv.IsMine)  
        {  
            m_grabbedCollider = grabPoint;  
            pv.RPC("SetKinematicTrue", RpcTarget.AllBuffered); //changes the kinematic state of the object to all players when its grabbed  
            if (onGrab != null)  
                onGrab.Invoke();  
            if (hideHandOnGrab)  
                m_grabbedBy.GetComponentInChildren<SkinnedMeshRenderer>().enabled = false;  
        }  
    }

    override public void GrabEnd(Vector3 linearVelocity, Vector3 angularVelocity)  
    {  
        //If the grabbed object is mine, release it  
        if (pv.IsMine)  
        {  
            rb.isKinematic = m_grabbedKinematic;  
            pv.RPC("SetKinematicFalse", RpcTarget.AllBuffered);  
            rb.velocity = linearVelocity;  
            rb.angularVelocity = angularVelocity;  
            if (hideHandOnGrab)  
                m_grabbedBy.GetComponentInChildren<SkinnedMeshRenderer>().enabled = true;  
            m_grabbedBy = null;  
            m_grabbedCollider = null;  
            if (onRelease != null)  
                onRelease.Invoke();  
        }  
    }

    public Collider[] grabPoints  
    {  
        get { return m_grabPoints; }  
        set { grabPoints = value; }  
    }

    virtual public void CustomGrabCollider(Collider[] collider)  
    {  
        m_grabPoints = collider;  
    }

    [Photon.Pun.PunRPC]  
    public void SetKinematicTrue()  
    {  
        gameObject.GetComponent<Rigidbody>().isKinematic = true;  
    }

    [Photon.Pun.PunRPC]  
    public void SetKinematicFalse()  
    {  
        rb.isKinematic = m_grabbedKinematic;  
    }  
}  

}

Matias
2020-11-25 12:57:59

Anyone?

Thomukas1
2020-12-01 12:53:44

I think it's really important to know that happens with your events "ongrab" and "onrelease". I am really intrigued that you are tackling this problem. Please consider joining my discord server https://discord.com/invite/G7eE9C9RRg I believe I can help you with this, if you give me more info.

emotitron
2020-12-02 02:35:45

There are some undocumented and not even complete implementations of oculus support in the Simple library (just got added to Pun2) in the SyncOculus folder you can poke around at.

You need to add the define OCULUS to the unity project though, and unpack the Oculus scene that is in the SyncOculus folder to get to it.

You can join our Discord to get some help messing with what I have going in that scene.
https://discord.gg/egaRfd8