Post-processing incoming streams in Photon.Voice.Unity.Speaker
In our app, we want to be able to do so post-processing of incoming audio streams, and since we'd previously hooked into Photon.Voice.Unity.Speaker.OnAudioFrame to read the stream, this seemed like a good place to manipulate the incoming audio samples. However, now periodically we're getting total dropout of the audio from one or more participants. It seems to affect participants on links with more latency more often, and we've also seen logs like the attached, which seems to originate from the same place.
Firstly, is modifying the samples in Speaker.OnAudioFrame supported? If so, is there a sample I can look at to make sure I'm doing any setup required (e.g. locking?). Or if not, is there a supported strategy for post-processing incoming audio before it is passed through to Unity's processing?
Best Answer
-
I assume you use the latest Voice API v2.50.
Do you mean it works correctly but if you add post-processing, the error shows up? The buffer size is wrong. This should not happen if you just modify the buffer contents.
Firstly, is modifying the samples in Speaker.OnAudioFrame supported?
No, but we could make it supported by changing this method to virtual. Then you could extend Speaker class and override the method with something like:
process(frame);
base.OnAudioFrame(frame);
There is also another approach that bypasses the Speaker and consumes packets directly form the stream via remoteVoiceLink.FloatFrameDecoded. See SaveIncomingStreamToFile.cs
But I don't think all of this is required since OnAudioFilterRead() exists. It seems like it was designed exactly for your case.
0
Answers
-
I assume you use the latest Voice API v2.50.
Do you mean it works correctly but if you add post-processing, the error shows up? The buffer size is wrong. This should not happen if you just modify the buffer contents.
Firstly, is modifying the samples in Speaker.OnAudioFrame supported?
No, but we could make it supported by changing this method to virtual. Then you could extend Speaker class and override the method with something like:
process(frame);
base.OnAudioFrame(frame);
There is also another approach that bypasses the Speaker and consumes packets directly form the stream via remoteVoiceLink.FloatFrameDecoded. See SaveIncomingStreamToFile.cs
But I don't think all of this is required since OnAudioFilterRead() exists. It seems like it was designed exactly for your case.
0 -
OK, I'll look into OnAudioFilterRead.
Right now we're directly patching the code in Speaker.cs to hook into OnAudioFrame, but the question was more about whether the buffer we see is being used on some other thread while we're modifying it.
0