OVRAvatar Expressive Features

I have PUN2 and Voice for PUN2 working with OVRAvatar. I'm struggling trying to get the Expressive Features working. I am aware of the single process issue with the microphone and Android, but have not been able to find the answer. I have found documentation mentioning "Voip.SetMicrophoneFilterCallback(MicFilter)" but I don't think this is relevant.

All I need is a way to access the PCM data recorded by Voice for PUN2 so I can send it to the OVRAvatar method "UpdateVoiceData". Can anyone help me find this data even if it means customizing PUN code?

Thanks!

Best Answer

  • JohnTube
    JohnTube ✭✭✭✭✭
    edited November 2021 Answer ✓

    Hi @blee808,

    Thank you for choosing Photon!

    If you want this from recording / sender client: create a post processor MonBehaviour ('has a' or 'is a' IProcessor<float>, Unity messages: PhotonVoiceCreated & PhotonVoiceRemoved, see SaveOutgoingStreamToFile.cs as example), add it to the same GameObject as a working Recorder component and call UpdateVoiceData(float[],int) inside Process method there (number of channels should be 1 mono or 2 stereo).

    If you want this from playback / receiver client: create a MonoBehaviour with Unity's OnAudioFilterRead method, add it to the same GameObject as working Speaker/AudioSource components and call UpdateVoiceData(float[],int) there. Or do something like in SaveIncomingStreamToFile.cs (call UpdateVoiceData(float[],int) inside remoteVoiceLink.FloatFrameDecoded, number of channels should remoteVoiceLink.VoiceInfo.Channels).

Answers

  • JohnTube
    JohnTube ✭✭✭✭✭
    edited November 2021 Answer ✓

    Hi @blee808,

    Thank you for choosing Photon!

    If you want this from recording / sender client: create a post processor MonBehaviour ('has a' or 'is a' IProcessor<float>, Unity messages: PhotonVoiceCreated & PhotonVoiceRemoved, see SaveOutgoingStreamToFile.cs as example), add it to the same GameObject as a working Recorder component and call UpdateVoiceData(float[],int) inside Process method there (number of channels should be 1 mono or 2 stereo).

    If you want this from playback / receiver client: create a MonoBehaviour with Unity's OnAudioFilterRead method, add it to the same GameObject as working Speaker/AudioSource components and call UpdateVoiceData(float[],int) there. Or do something like in SaveIncomingStreamToFile.cs (call UpdateVoiceData(float[],int) inside remoteVoiceLink.FloatFrameDecoded, number of channels should remoteVoiceLink.VoiceInfo.Channels).

  • Woohoo! Thank you for the answer. I was successful in getting lip sync working. Thank you for pointing out the IProcessor class and the SaveOutgoingStreamToFile example.

  • JohnTube
    JohnTube ✭✭✭✭✭

    Hi @blee808,

    If possible share with others your solution here or as gist / pastebin or GitHub / Bitbucket / GitLab repo so others can benefit from it.

  • Hey there @JohnTube, sure thing. I created a GitHub repo with a complete Photon & OVRAvatar solution which can be found here: https://github.com/octopus8/PhotonOVRAvatar

    I may have detected an issue with Voice for PUN2, though. When using the example provided above, when the second person joins the room, I get an invalid cast exception. Everything works fine, but the error is annoying. Any chance you could take a look and see if this is a bug?

    Thanks again, and I hope the project is helpful for the community.

  • JohnTube
    JohnTube ✭✭✭✭✭

    Hi @blee808,

    Thanks a lot for taking the time to do this!

    However, I think adding source code to repository instead of package only would have been more convenient and conventional. One step less to access / browse code from the web directly.

    The InvalidCastException is a kind of a known issue unless this is different from what I have in mind. I remember there was some InvaidCastException related to Oculus OVR due to some kind of 'race condition', @cloud_canvas was helpful at the time in fixing those. (see here for example)

    or maybe it's just another InvalidCastException thrown by PUN here and there maybe due to bad user code or maybe it's an edge case bug.

    You could post the full stacktrace from the logs so we have a better idea.

  • Ok, I checked in the code itself. Hope that is more helpful.

    Thanks for pointing out that other post. I'll see if I can get rid of the error using that suggestion. Here's the stacktrace for the error:

    Caught exception in OnEvent() for event code 253: System.InvalidCastException: Specified cast is not valid.

    UnityEngine.Debug:LogError (object)

    Photon.Realtime.LoadBalancingClient:DebugReturn (ExitGames.Client.Photon.DebugLevel,string) (at Assets/Photon/PhotonRealtime/Code/LoadBalancingClient.cs:2564)

    ExitGames.Client.Photon.PeerBase/<>c__DisplayClass108_0:<EnqueueDebugReturn>b__0 () (at D:/Dev/Work/photon-dotnet-sdk/PhotonDotNet/PeerBase.cs:1212)

    ExitGames.Client.Photon.EnetPeer:DispatchIncomingCommands () (at D:/Dev/Work/photon-dotnet-sdk/PhotonDotNet/EnetPeer.cs:440)

    ExitGames.Client.Photon.PhotonPeer:DispatchIncomingCommands () (at D:/Dev/Work/photon-dotnet-sdk/PhotonDotNet/PhotonPeer.cs:1771)

    Photon.Pun.PhotonHandler:Dispatch () (at Assets/Photon/PhotonUnityNetworking/Code/PhotonHandler.cs:221)

    Photon.Pun.PhotonHandler:FixedUpdate () (at Assets/Photon/PhotonUnityNetworking/Code/PhotonHandler.cs:147)