Photon Voice 2 and Voice Handler integration

I've been attempting to use Photon Voice 2 with the avatars and its Voice Handler (lipsync). I ran into the problem I saw a lot about with other lipsync assets and came to the workaround of using the debug echo in order for the lipsync to detect the microphone.

The Voice Handler on avatars automatically make the audio source mute and loop. I unmuted and turn off loop using RPC with OthersBuffered. Here's a breakdown of what happens:

  1. Player 1 joins and it works as intended
  2. Player 2 joins and it works as intended for Player 2, but Player 1 gets muted and loop is enable again.

My scene is set up as the documentation requires with the PhotonVoiceNetwork and the Recorder in the scene. The player prefabs are Instantiated with the Speaker as a child of the prefab as well as the Voice Handler.

Unity Version: 2020.3.25f1

PUN2 Version: 2.40

Photon Voice 2 Version: 2.31

I've been trying to fix this for a while and maybe I'm just not understanding something correctly. Any help would be appreciated.