Meta Avatars lip-sync

So I'm trying to get Avatars working, and for the most part it does, using both Photon Realtime and Photon Voice. The one issue I'm still struggling to solve is that the lip-sync is off. That is, the audio is being received/played back slightly out-of-sync with the avatar mesh animation that includes the lip movement blend shapes (sometimes by as little as a few ms, but sometimes as much as several hundred ms). I don't actually know/believe that this is a Photon problem, since both pieces are working as expected. But I'm wondering if there's a Photon solution. Is there some way to embed timestamps in both streams (I know there is in a custom packet with Realtime), and have the engine synchronize them in some way?

Thanks in advance!

Answers

  • If I understand correctly, you send lip sync animation keyframes via Realtime client and want to sync them with Voice client stream. Voice does not provide any sync tracks. Also, it can play the incoming stream faster to catch up if there was a delay in packets delivery. You can try to use Speaker.Lag property to estimate how much delay is added by playback. In theory, if the Lag is 200 ms when you are in perfect sync, you need to delay your animation by 100 ms if the Lag goes to 300,