Oculus Quest 2 with hand tracking, grabbable objects not synced.
The whole answer can be found below.
Try Our
Documentation
Please check if you can find an answer in our extensive documentation on PUN.
Join Us
on Discord
Meet and talk to our staff and the entire Photon-Community via Discord.
Read More on
Stack Overflow
Find more information on Stack Overflow (for Circle members only).
Oculus Quest 2 with hand tracking, grabbable objects not synced.
Matias
2020-11-24 14:45:39
Hi guys, I'm working in VR with the Oculus Quest 2 using Unity 2019.4.14f1 LTS version + PUN 2.
I managed to make hand tracking work and send the hands position and rotation through the network by using this plugin and the help of the developer. The hand's movement works really well without any lag or major latency problems.
Now... I'm having problems related to the synchronization of grabbable objects.
Let's say player1 and player2 are sharing a room. In front of them, there's a cube. If player1 grabs the cube and moves It with his hand, he will see the cube moving around without any problem, but for player2 the cube will instantly disappear as soon as player1 pinch the cube and move it. The cube will be visible for player2 again, only if player1 releases the object, which will suddenly appear in the position where player1 stopped grabbing the cube.
Everything else is working really well, the hand tracking, the avatars, the objects, voice, chat, etc.
I'm using modifications of the OVR grabber and OVR grabbable Oculus SDK scripts.
This is how the components set up for the grabbable object (a cube).
https://ibb.co/drJtF8P
And this is the components set up for the hand prefab I'm using.
https://ibb.co/SPS16wJ
Do you have any clue about this?
Comments
This is my custom PunOVRGrabber.cs script.
using Photon.Pun;
using System;
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
namespace Networking.Pun2
{
[RequireComponent (typeof (PhotonView))]
public class PunOVRGrabber : OVRGrabber
{
PhotonView pv;
//HandTracking related
PunOVRHand trackingHand;
private float pinchThreshold = 0.7f;
protected override void Awake()
{
base.Awake();
pv = GetComponent<PhotonView>();
if(GetComponent<PunOVRHand>() != null)
{
trackingHand = GetComponent<PunOVRHand>();
}
}
//Basically, if photonview is mine, update anchors from Grabber
public override void Update()
{
if (pv.IsMine)
{
base.Update();
//If hand tracking component detected, check pinch for grabbing mechanism.
if(trackingHand != null)
{
CheckPinch();
}
}
}
//If the pinch strenght is bigger than the threshold, call GrabBegin(), if smaller, call GrabEnd().
private void CheckPinch()
{
float pinchStrenght = trackingHand.GetFingerPinchStrength(PunOVRHand.HandFinger.Index);
if(!m_grabbedObj && pinchStrenght > pinchThreshold && m_grabCandidates.Count > 0)
{
GrabBegin();
Debug.Log("Grab begin");
}
else if (m_grabbedObj && (pinchStrenght < pinchThreshold))
{
GrabEnd();
}
}
}
}
And this is my custom OvrGrabbable.cs script:
using UnityEngine;
using UnityEngine.Events;
using Photon.Pun;
//
//Custom grabbable script which checks if the grabber "is mine" to actually grab
//
namespace Networking.Pun2
{
[RequireComponent(typeof(PhotonView))]
public class PunOVRGrabbable : OVRGrabbable
{
public UnityEvent onGrab;
public UnityEvent onRelease;
[SerializeField] bool hideHandOnGrab;
PhotonView pv;
Rigidbody rb;
protected override void Start()
{
base.Start();
pv = GetComponent<PhotonView>();
rb = gameObject.GetComponent<Rigidbody>();
}
override public void GrabBegin(OVRGrabber hand, Collider grabPoint)
{
m_grabbedBy = hand;
pv.TransferOwnership(PhotonNetwork.LocalPlayer);
if (pv.IsMine)
{
m_grabbedCollider = grabPoint;
pv.RPC("SetKinematicTrue", RpcTarget.AllBuffered); //changes the kinematic state of the object to all players when its grabbed
if (onGrab != null)
onGrab.Invoke();
if (hideHandOnGrab)
m_grabbedBy.GetComponentInChildren<SkinnedMeshRenderer>().enabled = false;
}
}
override public void GrabEnd(Vector3 linearVelocity, Vector3 angularVelocity)
{
//If the grabbed object is mine, release it
if (pv.IsMine)
{
rb.isKinematic = m_grabbedKinematic;
pv.RPC("SetKinematicFalse", RpcTarget.AllBuffered);
rb.velocity = linearVelocity;
rb.angularVelocity = angularVelocity;
if (hideHandOnGrab)
m_grabbedBy.GetComponentInChildren<SkinnedMeshRenderer>().enabled = true;
m_grabbedBy = null;
m_grabbedCollider = null;
if (onRelease != null)
onRelease.Invoke();
}
}
public Collider[] grabPoints
{
get { return m_grabPoints; }
set { grabPoints = value; }
}
virtual public void CustomGrabCollider(Collider[] collider)
{
m_grabPoints = collider;
}
[Photon.Pun.PunRPC]
public void SetKinematicTrue()
{
gameObject.GetComponent<Rigidbody>().isKinematic = true;
}
[Photon.Pun.PunRPC]
public void SetKinematicFalse()
{
rb.isKinematic = m_grabbedKinematic;
}
}
}
Anyone?
I think it's really important to know that happens with your events "ongrab" and "onrelease". I am really intrigued that you are tackling this problem. Please consider joining my discord server https://discord.com/invite/G7eE9C9RRg I believe I can help you with this, if you give me more info.
There are some undocumented and not even complete implementations of oculus support in the Simple library (just got added to Pun2) in the SyncOculus folder you can poke around at.
You need to add the define OCULUS to the unity project though, and unpack the Oculus scene that is in the SyncOculus folder to get to it.
You can join our Discord to get some help messing with what I have going in that scene.
https://discord.gg/egaRfd8