Meta Avatars lip-sync

Options
BlueSkyUnityGuy
BlueSkyUnityGuy ✭
in Photon Voice

So I'm trying to get Avatars working, and for the most part it does, using both Photon Realtime and Photon Voice. The one issue I'm still struggling to solve is that the lip-sync is off. That is, the audio is being received/played back slightly out-of-sync with the avatar mesh animation that includes the lip movement blend shapes (sometimes by as little as a few ms, but sometimes as much as several hundred ms). I don't actually know/believe that this is a Photon problem, since both pieces are working as expected. But I'm wondering if there's a Photon solution. Is there some way to embed timestamps in both streams (I know there is in a custom packet with Realtime), and have the engine synchronize them in some way?

Thanks in advance!

Answers

  • vadim
    vadim mod
    Options

    If I understand correctly, you send lip sync animation keyframes via Realtime client and want to sync them with Voice client stream. Voice does not provide any sync tracks. Also, it can play the incoming stream faster to catch up if there was a delay in packets delivery. You can try to use Speaker.Lag property to estimate how much delay is added by playback. In theory, if the Lag is 200 ms when you are in perfect sync, you need to delay your animation by 100 ms if the Lag goes to 300,

Cookies,
anyone?

We use cookies and related technologies to enhance your experience, show you personalised content and analyze performance and traffic on our website. By clicking on the „Accept All“ button you consent to the use of non-functional cookies and the subsequent processing of personal data to optimize our website and services as described in more detail in our Privacy and Cookie Policy.

By clicking on the „Customize or Deny all“ button you can decide otherwise. Clicking on the „Customize“ button will take you to a page where you can configure the usage of non-functional cookies (and related technologies) or deny all of them. You can access these settings at any time and also subsequently deselect cookies at any time in the footer area of our website.

ACCEPT ALL

Cookie Overview

We use the following categories of cookies and related technologies to enhance your experience, show you personalized content, and analyze performance and traffic on our website. We respect your right to privacy and accordingly you can chose to not allow some types of cookies (and related technologies). Click on the different category sliders and change our default settings to manage your cookie settings.

For more information on the specific cookies/related technologies we use and on how we use these, please see section 15 E. of our Privacy and Cookie Policy.

ABSOLUTELY NECESSARY

Authentication cookies we use are required to run our services … Cookies are required: Link

FUNCTIONAL AND MARKETING COOKIES

These cookies collect anonymous data and allow us to optimize our website and user experience. These cookies are listed here:

ANALYTICS

Help us to understand how visitors interact with our services, enables us to analyze and improve our services (also through third party analytics).