Photon Voice with SpeechToText library

DebugDotLog
edited November 2020 in Photon Voice
Hello there,
So I've been using Photon Voice and Photon Network for one of my project.
Recently I added a speech to text library and that's where troubles started.
There seems to be a conflict between PV and this library as they both try to get the mic input on Oculus Quest.
Everything works fine on PC though which is very strange.
Do you guys have any clue on a work around or what is the problem?
I've tried to "disable" one when the other is "enable" but that doesn't work on Quest either. Maybe it is just my way of doing it that is wrong.
As a use case example, when I use the speech to text API while being offline, it works. Then I go online and offline, and try again and it doesn't work anymore. And even when I quit the app and relaunch it, the speech to text library still doesn't work even offline. So I need to deinstall and reinstall the whole app for it to work again..

Thanks in advance and have a good day !

Comments

  • JohnTube
    JohnTube ✭✭✭✭✭
    Hi @DebugDotLog,

    Thank you for choosing Photon and sorry for the delay on this one.

    Did you check this?
  • DebugDotLog
    edited November 2020
    Hello @JohnTube yes I did un-check the "Can Own Microphone" check box.
    I've managed to make it works by setting the PhotonVoiceView.RecorderInUse.IsRecording to true whenever I'm creating a voice avatar and it's working fine.
    But now I currently have to disable the speechToText feature when I'm online cause it's not working when I'm voice recording at the same time on Quest (does work on windows computer though).
    I've tried to PhotonVoiceView.RecorderInUse.IsRecording to false whenever I use the SpeechToText feature and set it back to true when I'm done but it doesn't work on Quest either (this works on windows too btw).

    I have two issues at this point :
    - the first one is that this workaround works fine on Quest but I would like to know if I can use the speechToText feature while my RecorderInUse is recording. Would that create a conflict in your opinion or what would be your way to do such a thing ? I guess I might not do things as optimal regarding the PV integration. (Tell me if you need the coding part of my PV integration)

    - the second is that, I have 2 Unity scenes in my application. When I create a room as a Master and a client join the room (while I'm still in the first Unity scene) both Audio Avatar are sync and we can hear each other fine but when I create a room as a Master, then switch to the second Unity scene (while I'm the only one in the room) and then a client is joining, the client is in the right unity scene but only the master can hear the client and the client cannot hear the master as the master's audio avatar is not sync anymore (at least not instantiate in the client side). I did use PhotonNetwork.LoadLevel to switch scenes and I did set the photonNetwork.AutomaticallySyncScene to true on both ends. This is such a strange behavior as everything else (visuals, transforms etc) are sync correctly which means they are in the same room.

    Thank you for your time
  • JohnTube
    JohnTube ✭✭✭✭✭
    Hi @DebugDotLog,

    Maybe you did not have Recorder.AutoStart set to true and that's why you had to explicitly start recording?

    - issue 1: if the speech to text logic/code/plugin requires access to the microphone then yes it might be a problem/conflict between Photon Voice and another plugin trying to access the same microphone. If the speech to text plugin can accept the recorded audio data as input then you might want to make a custom Photon Voice processor attached to the same Recorder component. Get some hints here (check second part for outgoing streams).
    - issue 2: make sure there is only one PhotonVoiceNetwork instance in the hierarchy and that one instance is marked as DontDestroyOnLoad. This singleton pattern is important. Besides, make sure everything else works fine, you could check the in Editor debug info in PhotonVoiceNetwork custom inspector (foldouts) or make use of this VoiceDebugScript, attached to the same GameObject as PhotonVoiceView and then call "VoiceDebugScript.CantHearYou" for the person you can't hear.
  • Hello @JohnTube ,

    I did have the Recorder.AutoStart set to true which is strange.

    -issue 1 : Sounds awesome, I'll try to implement this next week and I'll let you know !

    -issue 2: So there is only one PhotonVoiceNetwork instance in the hierarchy and ofc its marked as DontDestroyOnLoad.
    The think is, everything works fine as long as I join the master in the first scene (which is the login/lobby scene).
    To be more specific the problem is, when I join a room I automatically PhotonNetwork.Instantiate an Avatar which has PhotonView, PhotonVoiceView, Speaker, AudioSource and the VoiceDebugScript as components and set the PhotonVoiceNetwork gameObject as its parent (so I can keep my hierarchy organized).
    And so when I join the room as a client while the master is in the first scene I can see in the hierarchy two Avatar, one is the master's and the other the client's.
    But when I join the room (as a client) and the master is already in the second scene, I can only see the one avatar on the client side (which is the client avatar only). On the master side though I can see both. And so I can't run the CanHearMe? method on the right Avatar has there is no Avatar in the hierarchy (and that's probably why the client can't hear the master anyway).

    Thank you for your time
  • JohnTube
    JohnTube ✭✭✭✭✭
    Hi @DebugDotLog,

    issue 2:

    It looks like the avatar prefab clone is instantiated in the first scene and when the new scene is loaded it got removed with the previously unloaded first scene.
    So it's a timing issue, not sure how you load scenes? do you use PhotonNetwork.LoadScene?
  • DebugDotLog
    edited November 2020
    Hello @JohnTube ,

    I use this part of code :
    PhotonNetwork.OpRemoveCompleteCache();
    PhotonNetwork.LoadLevel(mySceneName);

    maybe it's the OpRemoveCompleteCache method that makes the issue, does it need to be call when loading other scene ? or only when the master leaves the room ?

    (on issue 2 is PhotonVoice doing anything special regarding the microphone configuration in awake or before anyone joins a room and therefor instantiate an avatar etc ?)