AudioOutCapture attaching to another audioListener

Philipp · November 2019

Hi! I'm trying to understand the implications of the AudioOutCapture component. When I have the WebRTCAudioDSP component, it during Awake looks for any AudioListener in the scene. My project happens to have a currently unused VR camera, which has an AudioListener so AudioOutCapture attaches to it, but that camera is inactive and is being replaced by an added proper In-person-head mobile camera later on once the Person Photon-instantiates (which happens after room-join).

Does AudioOutCapture being attached to this wrong camera cause any issues? (I need proper 3d spatial sound, so not sure if it interferes somehow.) I'm also trying to understand the relation between this AudioOutCapture and the already existing recorder, attached to the same object holding PhotonVoiceNetwork. On the surface, I don't seem to hear any issues. Thanks!

JohnTube · November 2019

Hi @Philipp,

here is a suggested quick fix for this newly encountered situation:

In "Assets\Photon\PhotonVoice\Code\WebRtcAudioDsp.cs", in WebRtcAudioDsp.Awake method, replace one line:

AudioListener audioListener = FindObjectOfType<AudioListener>();

with:

            AudioListener[] audioListeners = FindObjectsOfType<AudioListener>();
            AudioListener audioListener = null;
            for(int i=0; i < audioListeners.Length; i++)
            {
                if (audioListeners[i].gameObject.activeInHierarchy)
                {
                    audioListener = audioListeners[i];
                    break;
                }
            }

Let us know if this helps.
Thanks again for all your reports and your time!

Philipp · November 2019

Thank you for the help!

In my current situation, I have the special situation where the actual audio listener is not in the scene yet -- because it's part of our person that I'm still about to instantiate in a second later -- and on top, it may change (because the Person will be deinstantiated + reinstantiated upon scene change). I suppose for now I'll try to have the original audioListener simply be centerish and hope for the best. Maybe there could be a function like "UpdateAudioListener()" in WebRtcAudioDsp, called on need by devs? Would that work, to have it reassign in realtime?

Just to understand, does the position of the audioListener used by WebRtcAudioDsp determine the spatial audio sound feel that is then transmitted to other people in the room?

JohnTube · November 2019

You can only have one AudioListener in the scene.

The "effective active enabled AudioListener actually in use" is responsible for "the spatial audio sound feel" in 3D yes.

If you could tell me how you switch AudioListeners in your scenes or in the same scene at runtime by instantiating new one(s), maybe I can think about ways to handle this better.

Of course, a method to override or manually and explicitly set the AudioListener or the AudioOutCapture component to be used with the WebRtcAudioDsp is an option.

Philipp · November 2019

I only need 1 AudioListener, the problem is that my 1 AudioListener is only photonNetwork-instantiated with the Person object (after connecting to a room), which means -- as PhotonVoiceNetwork, Recorder and WebRTCAudioDSP are available right from the start -- that WebRTCAudioDSP will throw an "AudioListener component is required" error at startup. For that reason, I had added back a stand-in audioListener/ camera. But it then permanently attaches the AudioOutCapture to that wrong audioListener.

What's the proper setup flow for all of this? (E.g. person instantiating, where the camera is "inside" the instantiated person's head.) I might be doing it wrong. Thanks!

I might need to look again into having 1 persistent camera, which then dynamically moves into the person's head (instead of being in its prefab).

Philipp · November 2019

I'm now doing that above, having 1 persistent camera right from start, which then dynamically moves into ourPerson's head, which seems to solve my immediate problems. Still curious about what the normal workflow for all of this would be!

JohnTube · November 2019

Hi @Philipp,

AudioOutCapture is used to get the audio output necessary for acoustic echo cancellation.

I see.
What you did is OK but you can also:

A- instantiate Recorder + WebRtcAudioDsp at runtime post prefab instantiation == the new AudioListener is ready.
B- attach the Recorder + WebRtcAudioDsp to the prefab itself.
C- instantiate WebRtcAudioDsp post prefab instantiation. This requires either delay of recording until the new AudioListener is ready, i.e. do not start recording only until then (disable Recorder.AutoStart) OR restart recording.

Or you can try this code after adding it to the WebRtcAudioDsp:

public bool SetOrSwitchAudioListener(AudioListener audioListener)
        {
            if (audioListener == null)
            {
                if (this.Logger.IsErrorEnabled)
                {
                    this.Logger.LogError("audioListener passed is null or is being destroyed");
                }
                return false;
            }
            if (audioListener.gameObject.activeInHierarchy)
            {
                if (this.Logger.IsErrorEnabled)
                {
                    this.Logger.LogError("The GameObject to which the audioListener is attached is not active in hierarchy");
                }
                return false;
            }
            if (!audioListener.enabled)
            {
                if (this.Logger.IsErrorEnabled)
                {
                    this.Logger.LogError("audioListener passed is disabled");
                }
                return false;
            }
            AudioOutCapture audioOutCapture = audioListener.GetComponent<AudioOutCapture>();
            if (audioOutCapture == null)
            {
                audioOutCapture = audioListener.gameObject.AddComponent<AudioOutCapture>();
            }
            return this.SetOrSwitchAudioOutCapture(audioOutCapture, false);
        }

        private bool SetOrSwitchAudioOutCapture(AudioOutCapture audioOutCapture, bool checkAudioListener)
        {
            if (audioOutCapture == null)
            {
                if (this.Logger.IsErrorEnabled)
                {
                    this.Logger.LogError("audioOutCapture passed is null or is being destroyed");
                }
                return false;
            }
            if (!audioOutCapture.enabled)
            {
                if (this.Logger.IsErrorEnabled)
                {
                    this.Logger.LogError("audioOutCapture passed is disabled");
                }
                return false;
            }
            if (checkAudioListener)
            {
                if (audioOutCapture.gameObject.activeInHierarchy)
                {
                    if (this.Logger.IsErrorEnabled)
                    {
                        this.Logger.LogError("The GameObject to which the audioOutCapture is attached is not active in hierarchy");
                    }
                    return false;
                }
                AudioListener audioListener = audioOutCapture.GetComponent<AudioListener>();
                if (audioListener == null)
                {
                    if (this.Logger.IsErrorEnabled)
                    {
                        this.Logger.LogError("The AudioListener attached to the same GameObject as the audioOutCapture is null or being destroyed");
                    }
                    return false;
                }
                if (!audioListener.enabled)
                {
                    if (this.Logger.IsErrorEnabled)
                    {
                        this.Logger.LogError("The AudioListener attached to the same GameObject as the audioOutCapture is disabled");
                    }
                    return false;
                }
            }
            if (this.ac != null)
            {
                if (this.ac != audioOutCapture)
                {
                    if (this.started)
                    {
                        this.ac.OnAudioFrame -= this.OnAudioOutFrameFloat;
                    }
                }
                else
                {
                    if (this.Logger.IsErrorEnabled)
                    {
                        this.Logger.LogError("The same audioOutCapture is being used already");
                    }
                    return false;
                }
            }
            this.ac = audioOutCapture;
            if (this.started)
            {
                this.ac.OnAudioFrame += this.OnAudioOutFrameFloat;
            }
            return true;
        }

        public bool SetOrSwitchAudioOutCapture(AudioOutCapture audioOutCapture)
        {
            return this.SetOrSwitchAudioOutCapture(audioOutCapture, true);
        }

Philipp · November 2019

Thank you, that's very insightful!

I guess as it's used for echo cancellation, now that I move the AudioOutCapture into the actual person's head position, it might work even better at cancelling it (?).

JohnTube · November 2019

it might work even better at cancelling it (?).

I don't think it changes anything. Maybe, I did not test it before.

Philipp · November 2019

Right, thanks. I suppose the audio captures only what the current listener hears anyway (even when it switches), independent of where the audio capture component itself is attached. And my listener was already correct the last time. Well, it's cleaner now in any case, because I don't get any more Unity "There can only be 1 audioListener in the scene" alerts during switching areas!

AudioOutCapture attaching to another audioListener

Comments