Passing custom float[] buffer through Photon voice

Hello,

I am applying a voice filter to data I recieve from the microphone on the Unity side in realtime. Once all is completed I have a new float[] that I turn into an audio clip and play that back and I hear what I said with the filtered applied with some latency. I am now trying to get that float[] through Photon voice so others will hear it as well. Is there a best approach to this? I know you can change the Recorder to either use an AudioClip or InputFactory. Audio clip doesnt seem to be the way to go since I need to restart the mic every time I change the clip according to logs and would be problematic in realtime .

I tried InputFactory but it doesnt seem to be working. What I tried with input factory is copying AudioClipWrapper and applying the same logic to the newly created audio clip and I tried just setting the buffer in IAudioReader.Read(float[] buffer) to the buffer I used to create the audio clip. Both approached end up with hearing nothing on the Debug echo side.

Perhaps I am not setting the data correctly during the Read() part. Any ideas on how to approach this problem?


Thanks,

Jacob

Comments

  • Hi Jacob,

    Does InputFactory work with AudioClipWrapper if you just assign a clip to it as Recorder does? If yes, It should work also after updating the clip.

    if you have float[] at some point, you can send it directly through Photon voice via IAudioPusher<float> instance which InputFactory creates. You just need to store the callback in implementation of

     void SetCallback(Action<T[]> callback, ObjectFactory<T[], int> bufferFactory)

    Then call this callback on each float[] you produce.

    You can ignore bufferFactory or use it for your buffers to avoid unnecessary allocations.

  • Hello,

    Thanks for the reply.

    I tested out the IAudioPusher but it was very choppy.

    I also tested out the AudioClipWrapper with replacing the clip with the new one when it is made.

    This approach was not orginally working either but after setting the clip and then resetting startTime = Time.time and readPos = 0 seems to fixed it for the most part. The only issue now is that every so often there is a stutter in the audio where I hear nothing for about 0.5-2 seconds and it picks back up.

    Any idea why this would be? Some extra info is I am creating a clip in the update at an interval, at the moment its 0.5 so it should fire off every half second and putting the data from the microphone clip into a buffer accounting for looping etc. and then creating a clip with that buffer.

  • vadim
    vadim mod
    edited November 2022

    Hi,

    I tested out the IAudioPusher but it was very choppy.

    Did you implement SamplingRate and Channels properties correctly? They should match you audio signal parameters.

    You are trying to use AudioClipWrapper instead of proper implementation of IAudioReader/IAudioPusher. AudioClipWrapper is IAudioReader but designed not for you case. You do not need an audio clip to feed Voice with data.

  • Hello,

    I set the sampling rate in the recorder to 48000, the clip I create says its freq is 48000, the channels is telling me its 1, on my microphone.start where i get my info from is set to 48000 and I have this in my AudioPusher.

    public int SamplingRate => 48000;

    public int Channels => 1;

    When using this method my voice comes in for split second in the beginning before getting cutoff. I also tried skipping the clip creation part and just passing in the buffer I use to create the clip to the callback but it has the same results.