Salsa and the PUN Voice Demo

HipshotHipshot
edited February 14 in Photon Voice
I got salsa lip sync working with Photon partially. The lip sync works with remote clients, but not the local machine. This makes sense because the local client does not get its own audio - I am thinking to avoid delay echo effects.

On the remote clients, the Audio Source has as the audio clip "AudioStreamPlayer" - but locally it has "None (Audio Clip)"

My thinking was to add a microphone component to the local client (salsa provides one), to drive lip sync. This works, however not with the PhotonVoiceSpeaker script together.

I thought if I could deactivate the PhontVoiceSpeaker script on the local client, everything would be good. Locally the lip sync would be driven by the SALSA mic input, and on remote clients, it would be driven by the AudioStreamPlayer in the AudioSource.

My idea was to add "if(!IsPlaying)" code to the PhontVoiceSpeaker script :
if(!IsPlaying)
        this.player = new AudioStreamPlayer(GetComponent<AudioSource>(), "PUNVoice: PhotonVoiceSpeaker:", 

It seemed to work in the editor, but when I published the game to test multiplayer, no go.

Any insight is much appreciated.

Comments

  • Hi @Hipshot,

    Thank you for choosing Photon!

    What do you need exactly to integrate Photon Voice with Slasa Lip Sync?

    A possible (easiest) trick is to set DebugEchoMode to true and mute the AudioSource of the local player.
    this means that you will recieved your own voice from the server and play it in the AudioSource using PhotonVoiceSpeaker/AudioStreamPlayer but muted.

    A more advanced and proper solution would be to actually get the audio frames recorded (before or after) transmission locally and make the AudioClip (AudioStreamPlayer) out of them. You need to inject a pre / post processor in PhotonVoiceRecorder for this to work and follow what PhotonVoiceSpeaker does to received audio frames. See here (the part "Second, outgoing voice streams:").

    I see that you are using Photon Voice Classic, could you migrate to Photon Voice 2?
    This is unrelated but recommended.
    Photon Voice Classic will be phased out.
  • HipshotHipshot
    edited February 18
    Thanks for this answer! I was able to solve this in a bit of a different way, although your direction is better.


    my solution: https://forum.unity.com/threads/salsa-lipsync-suite-lip-sync-emote-head-eye-and-eyelid-control-system.242135/page-35#post-5490615

    As far as photon voice classic - our project built on this tech it and it is working well for our purposes, thanks much for continuing to support it. Next time for sure!
Sign In or Register to comment.