Using FMOD with Photon Voice in Unity

Bexelius
2022-02-21 15:52:02

I noticed that there were a FMOD folder under Assets/Photon/PhotonVoiceApi/Platforms, and tried to hook it up.

I have almost got it to work after these steps:

• Enable the define PHOTON_VOICE_FMOD_ENABLE

• Modify Speaker.cs to use a FMODUnity.StudioEventEmitter instead of an AudioSource (and Photon.Voice.FMOD.AudioOutEvent instead of Photon.Voice.Unity.UnityAudioOut).

• Make a looping programmer instrument event in FMOD (I made it 3 seconds long since that is how long buffer is created in AudioOut.cs)

So this is working and you can hear voice, but it keeps starting and stopping every 3 seconds, spamming this in the Console:

[PV] [FMOD] AudioOut (Event) EventCallback CREATE_PROGRAMMER_SOUND

[PV] [FMOD] AudioOut (Event) Sound Assigned to Event Parameter

[PV] [FMOD] AudioOut (Event) EventCallback SOUND_PLAYED

[PV] [FMOD] AudioOut (Event) EventCallback SOUND_STOPPED

[PV] [FMOD] AudioOut (Event) EventCallback DESTROY_PROGRAMMER_SOUND

[PV] [FMOD] AudioOut (Event) EventCallback CREATE_PROGRAMMER_SOUND

[PV] [FMOD] AudioOut (Event) Sound Assigned to Event Parameter

... (and so on) ...

Why is the sound stopped and recreated all the time?

Comments

Bexelius
2022-02-21 18:13:17

I solved it!

If anyone attempts this themselves, you need to make the programmer instrument both async and looping, inside an equally long looping region.

JohnTube
2022-02-22 06:18:38

Hi @Bexelius,

Thank you for choosing Photon and congratulations on making this work for you!

vadim
2022-02-22 09:06:07

Glad to hear that FMOD works for you. Are your sure that FMODUnity.StudioEventEmitter is required? Switching to another AudioOut implementation should be enough. In future SDK versions that would be possible without Speaker.cs modification, via factory as it is now for AudioIn.

Bexelius
2022-02-22 10:46:04

I need an event so we can control the fmod side of it (attenuarion, track for volume etc)

Bexelius
2022-02-22 15:34:14

I may have spoken too soon.

Fmod did work for me when I tested two clients running locally on PC, but I had to increase the programmer instrument to be as long as 15 seconds to make it work reliably.

However, when running two clients on Android, the sound is still cutting out for us about every 3 seconds (suspiciously similar to the 3 second long buffer created by AudioOutDelayControl).

Furthermore, we control the volume of the voice in game using a fmod VCA. If I drag that all the way down to 0 and then back up again, the sound gets all garbled.

Neither of these issues happened with the default Unity AudioSource output.

Any ideas?

Bexelius
2022-02-22 18:54:21

Some debugging showed that it was the fmod AudioInput that didnt work well on android. We will use Photon Microphone instead as a work around.

The volume issue seems to have been from having Envelope=Auto on the programmer instrument in fmod. Setting it to off solved the garbled sound issue.

vadim
2022-02-25 09:37:05

Thanks for sharing this.

Bexelius
2022-03-07 10:16:57

After testing for a few weeks, we still get some issues with this setup. Once in a while, the audio is turning out scrambled for one of the participants.

Any idea what could be causing this?

gorrj
2022-03-07 20:49:22

@Bexelius We are also trying to get this working and are encountering the same garbled audio issue. One person I can hear fine but they can't understand anything that I'm saying. We'll be trying to resolve this this week.

Bexelius
2022-03-07 20:58:46

@gorrj Did you check the Envelope setting on the programmer instrument in fmod studio? Setting that to off made volume work for us, though it still turns out scrambled randomly.

I reviewed the photon code and saw that the buffer used is delay max (1000ms default) * 3 (hardcoded). But the actual seek will be from a point modulo that length, and then so much longer forward, so Im thinking an instrument 2x that length would be needed (ie 6 seconds). Could use some official guideline on this though…

Bexelius
2022-03-08 15:52:22

I noticed when debugging this that the constructor in Photon.Voice.FMOD.AudioOut was using processInService: false for the base class constructor, while the old UnityAudioOut was using processInService: true. In other words, the fmod output used a code path that hasnt really been used before, and thus susceptible to bugs. I changed it to true for fmod too, and it still sounds good. Will try this for a while and see if it solves the issue...

gorrj
2022-03-08 16:01:10

@Bexelius Our programmer instrument did not have an envelope. We can play with the track length but it was already set to 15 seconds. Thank you for that observation.

Bexelius
2022-03-10 17:00:37

None of my attempts have helped to remove the occacional garbled sound. Furthermore, I also found an issue where creating new speakers to occacionally failed to create an fmod sound.

None of these issues happen when using UnityAudio, so we have decided to abandon this effort and revert to using the default UnityAudio output.

FlorianBrink
2022-04-04 08:58:32

Thank you for creating this thread @Bexelius! Any luck fixing the garbled sound issue so far? Is it worth spending time trying to implement?

This seems to literally be the onlyweb page about using Photon Voice with FMOD. Would you have any more information to share with us @JohnTube?

vadim
2022-04-06 09:43:15

FMOD support in Photon Voice is experimental. Its main purpose is to demonstrate integration of 3rd party libraries with the Voice. Both FMOD.AudioOut and FMOD.AudioOutEvent work fine in my tests. But maybe my setup is different from what real FMOD users have. My experience in FMOD is limited. I just made it working in my simple test.

Can you reproduce the issue with DemoVoiceUI or other demo from Photon Voice?

For FMOD.AudioOutEvent, make sure that:

• the event contains "Programmer Instrument"

• Programmer Instrument is looped

• the event is persistent

• probably the event should be assigned to a bank

• fmod project is specified in unity fmod settings

• don't forget to run "build" in fmod studio after modifications

FlorianBrink
2022-04-06 10:30:21

@vadim Thanks for your reply.

I'm getting errors right after enabling the PHOTON_VOICE_FMOD_ENABLEdefine:

• FMODAudioIn.cs(5,17): error CS0246: The type or namespace name 'FMOD' could not be found

• FMODAudioInEnumerator.cs(4,17): error CS0246: The type or namespace name 'FMOD' could not be found

• FMODAudioOut.cs(5,17): error CS0246: The type or namespace name 'FMOD' could not be found

Rider shows FMOD in red here:

using FMODLib = FMOD;    

The FMOD plugin is installed in the project and works as expected. I am not a namespace person and I couldn't figure it out. Any idea? Thanks.

FlorianBrink
2022-04-06 10:33:38

Well that didn't take long. Immediately after sending the above message, I tried to add FMODUnity to PhotonVoice.API asmdef and that made the trick.

vadim
2022-04-08 12:04:52

References to 3rd party dependencies may be missing or invalid in the current Photon Voice package. We will look into this.

JMota
2023-04-25 12:19:39

I'm trying to set this with no luck.

Do you have an example of this working?

Thanks!

vadim
2023-04-26 09:13:39

What are you trying to set up and what results or errors do you get?

To work with FMOD, you need to define PHOTON_VOICE_FMOD_ENABLE. Then SpeakerFMOD component becomes available. Use it instead of Speaker to play audio via FMOD.

JMota
2023-04-26 10:02:30

Yep, I set the define symbols, I've replaced the AudioSource by **FMOD Studio Event Emitter** and **Photon.Voice.FMOD.AudioOutEvent** instead of Photon.Voice.Unity.UnityAudioOut. I don't know if I'm setting the values of the UnityAudioOut properly or if I'm setting "looping programmer instrument" event in FMOD wrongly.

I've used the DemoVoiceUIsample scene to manage this. In the Speaker.cs I've replaced this line

//return new UnityAudioOut(this.GetComponent<AudioSource>(), this.playDelayConfig, this.Logger, string.Empty, true);

by this one:

 ``**return new Voice.FMOD.AudioOutEvent<float>(FMODUnity.RuntimeManager.CoreSystem, instance, this.playDelayConfig, this.Logger, string.Empty, true);**

and all the AudioSource by **FMODUnity.StudioEventEmitter**.

In the looping programmer instrument, I've created into an event in FMOD Studio and I've linked that into an event that is assigned into an FMOD Studio Event Emitter

I cannot listen to anything in the "DebugEcho" option into the scene. The final aim of this is to apply pitch shifting to the photon voice audio without altering the tempo, and to achieve this I'm doing it through FMOD. But I'm not sure what I'm doing wrong there.

JMota
2023-04-26 10:17:19

Yep, I've set the defined symbols.

I've modified the Speaker.cs file doing the following:

• Replace AudioOutEvent:

      //return new UnityAudioOut(this.GetComponent<AudioSource>(), this.playDelayConfig, this.Logger, string.Empty, true);

by this one:

       ``**return new Voice.FMOD.AudioOutEvent<float>(FMODUnity.RuntimeManager.CoreSystem, instance, this.playDelayConfig, this.Logger, string.Empty, true);**

Replacing every AudioSource by **FMOD Studio Event Emitters**

• :

Make a looping programmer instrument event

• in FMOD (I made it 3 seconds long since that is how long buffer is created in AudioOut.cs). Probably this is one of the possible mistakes, I'm not sure if this goes in this way.

The thing is, I'm trying to reproduce the voice from Photon Voice using the "DebugEcho" option, but it doesn't work when using the FMOD Studio Event Emitters. The aim of doing this is to finally apply a pitch shifting effect using FMOD without altering the tempo of the audio, but at this moment, I can't make it work just using the normal voice.

vadim
2023-04-26 12:46:25

Please use SpeakerFMOD and try it in non-event mode first.

JMota7
2023-05-23 11:15:20

ED: Sorry about the duplicated comments 😅

JMota7
2023-05-23 11:15:37

Sorry,

I used SpeakerFMOD and apparently it was reproducing the audio, also, it was applying some pitch shifter, but... I'm receiving these errors:

[FMOD] ShadowEventInstance::createProgrammerSound : Programmer sound callback is not set for instrument ''.

[FMOD] EventInstance::createProgrammerSoundImpl : Programmer sound callback for instrument '' returned no sound.

Apparently, the programmer instrument is not receiving the audio. Now, I'm using the SpeakerFMOD instead of the Speaker. The audio seems to be played, but it seems that I can't change the pitch shifter value (it changes the value just if I change the pitch in FMOD before starting the demo app but not on runtime), because it says the programmer sound callback for instrument returned no sound.

Any leads here?

Thanks!