Output all voice conversations into .wav, .mp3 etc

Hi,

I will be required to make a VR serious game that involves voice commands between the instructor and student as part of the training.

Will it be possible to record voice conversations using the plugin?

I needed to log them for review next time.

Thanks

Comments

  • vadim
    vadim mod
    edited August 2017
    Hi,
    Unity provides MonoBehaviour.OnAudioFilterRead which can be used to capture incoming audio per remote voice PhotonVoiceSpeaker (AudioSource) or record full application output mix if implemented on object with AudioListener.
    For outgoing audio stream, you can create Voice.LocalVoiceAudioFloat.IProcessor and insert it in local voice processing pipeline. See DelayProcessor.cs
  • @vadim ,
    Hi Guys,

    I have a multiuser application which runs with Photon Unity Network (PUN) and Photon Audio. With Photon Audio, audio is streamed to the PUN server and then sent to all the players in the game. Photon Audio has a wrapper class on top of the core Unity audio classes to do their streaming. I would into implement in my application:

    1. record to a wav file the incoming audio streaming from the Photon Audio server,
    and
    2. redirect the Photon Audio streaming to another service feed.

    I have a script to record and save the locally produced audio before it is streamed. I have read about the MonoBehaviour.OnAudioFilterRead, but I do not quite understand how could I use it to record the incoming audio stream.

    Any guidance would be greatly appreciated.
  • Hi,
    You can add incoming streams handlers during app initialization like this:
    PhotonVoiceNetwork.Client.OnAudioFrameAction += (playerId, voiceId, frame) => Debug.LogFormat("{0} {1} {2}", playerId, voiceId, frame[0]);
    playerId and voiceId uniquely identify stream, frame is a float[] containing audio data.
  • vadim said:

    Hi,
    You can add incoming streams handlers during app initialization like this:
    PhotonVoiceNetwork.Client.OnAudioFrameAction += (playerId, voiceId, frame) => Debug.LogFormat("{0} {1} {2}", playerId, voiceId, frame[0]);
    playerId and voiceId uniquely identify stream, frame is a float[] containing audio data.

    Thank you @vadim , for your reply. Do you think would it be a sensible approach to tap into the file Photon Voice Speaker and use the parameter passed on OnVoiceLinked(int frequency, int channels, int frameSamplesPerChannel, int playDelayMs) and on OnAudioFrame(float[] frame) to create and save the wav file from there?
  • Why PhotonVoiceNetwork.Client.OnAudioFrameAction does not work for you?
    I forgot to mention PhotonVoiceNetwork.Client.OnRemoteVoiceInfoAction which provides info on new remote voice and PhotonVoiceNetwork.OnRemoteVoiceRemoveAction signalling remote voice removal.

    Of course you can modify you copy of PUNVoice sources but it makes harder to update to new PUNVoice versions.
  • @vadim, Thank you so much. The update argument is quite on point. I will work with PhotonVoiceNetwork.Client.OnAudioFrameAction.
  • @vadim , I am having trouble figuring out how to use the float value returned by PhotonVoiceNetwork.Client.OnAudioFrameAction. Could you please lead me on how to store this stream? I think I need to store it before creating the wav file. Thanks in advance.
  • Towerss
    Towerss
    edited May 2018
    @vadim , If you don't mind having a look, this is what I have till now:
        public void SaveAudioStream()
        {
    
            string filename = "TestWavFile";
    
            if (!filename.ToLower().EndsWith(".wav"))
            {
                filename += ".wav";
            }
    
            //var filepath = filename;
            var filepath = Path.Combine(Application.persistentDataPath, filename);
    
            Debug.Log("*** Filepath: " + filepath);
    
    
            // Make sure directory exists
            Directory.CreateDirectory(Path.GetDirectoryName(filepath));
    
    
            FileStream writeStream = new FileStream(filepath, FileMode.Create, FileAccess.Write);
            PhotonVoiceNetwork.Client.OnAudioFrameAction += (playerId, voiceId, frame) => Debug.LogFormat("***** {0} {1} {2}", playerId, voiceId, frame[0]);
    
            //I need to feed the MemoryStream buffer with the Audio data returned from PhotonVoiceNetwork.Client.OnAudioFrameAction
            MemoryStream memstrm = new MemoryStream();
    
            int Length = 256;
            Byte[] buffer = new Byte[Length];
            int bytesRead = memstrm.Read(buffer, 0, Length);
            // write the required bytes
            while (bytesRead > 0)
            {
                writeStream.Write(buffer, 0, bytesRead);
                bytesRead = memstrm.Read(buffer, 0, Length);
            }
    
        }
  • The code does not make sense. You set handler which simply logs on each frame. Not sure what code below does.
    In case you want to capture entire stream, open file in OnRemoteVoiceInfoAction handler, write frame of audio data in OnAudioFrameAction handler and close the file in OnRemoteVoiceRemoveAction.
    Or you can open and close file and update OnAudioFrameAction accordingly on user input.
  • Towerss
    Towerss
    edited July 2018
    Hi @vamin, It's me again. I would like to have your help with another recording related issue.

    The first approach, I was using the OnAudioFrameAction handler to get all the frames received and save them in a wav file. I am using an "observer" instance that joins the session but only to record audio. That worked well for a single client connected apart from the "observer". As soon as I got two or more clients connected, the audio file was recording the float frames from the different clients overlapped, making the wav file useless.

    The second approach, I was using MonoBehaviour.OnAudioFilterRead to get the frames on PhotonVoiceSpeaker.cs and the problem here is that I am having separate files for each client connected, not the whole conversation. The reason why I used this file is that my application says I already have an AudioListener somewhere, but I could not find it.

    The third approach was using AudioStreamPlayer.cs without MonoBehaviour.OnAudioFilterRead. It worked the same way as in the second approach: there is a file opened for every client connected.

    I would appreciate if you could give any other hint about how to record the whole incoming conversation in a single file.

    Thank you.
  • Hi @vamin, no worries, all sorted now.

    In the application I was debugging, there was a combination of [HideInInspector], HideFlags.HideInHierarchy, and HideFlags.HideInInspector applied to some gameobjects and components from third-party libraries.

    I found the required gameobjects and components, and I used your suggested approach to use MonoBehaviour.OnAudioFilterRead with the gameobject that instantiates the AudioListener and everything is working now. I have one single WAV file with the whole conversation in good quality :)

    Thank you.
  • RK_Rathod042
    edited September 2021
    Towerss wrote: »
    Hi @vamin, no worries, all sorted now.

    In the application I was debugging, there was a combination of [HideInInspector], HideFlags.HideInHierarchy, and HideFlags.HideInInspector applied to some gameobjects and components from third-party libraries.

    I found the required gameobjects and components, and I used your suggested approach to use MonoBehaviour.OnAudioFilterRead with the gameobject that instantiates the AudioListener and everything is working now. I have one single WAV file with the whole conversation in good quality :)

    Thank you.

    - hey, Are those methods you are mention above are working on v2 ?
    - Can you please post your code ? because I don't found the "PhotonVoiceNetwork.Client.OnAudioFrameAction" class methods
    - please help me-
    - now I have completed saving outgoing audio but it's audio quilty is very bad-
    - Check code here -> https://github.com/rk042/PhotonPunVoice/blob/main/Assets/Scenes/SaveOutVoice.cs
  • Hi @RK_Rathod042, I responded to your message. I do not have the whole code snipped, but this SO answer is the base for what I used: https://stackoverflow.com/questions/50302315/save-audio-stream-float-frames-as-wav-with-c-sharp
  • JohnTube
    JohnTube ✭✭✭✭✭
    edited September 2021
    Hi @Towerss, @RK_Rathod042,

    Thank you for choosing Photon!

    Photon Voice 2 contains two utility scripts to save incoming and outgoing audio streams to wav files.
    Simply add "Assets\Photon\PhotonVoice\Code\UtilityScripts\SaveIncomingStreamToFile" to the same GameObject as VoiceConnection (or PhotonVoiceNetwork) or "Assets\Photon\PhotonVoice\Code\UtilityScripts\SaveOutgoingStreamToFile" to the same GameObject as Recorder.
    Files are saved to Unity's persistent data folder.
  • @JohnTube thanks, there are working but the audio quality is very poor.
  • Thanks for sharing.