New pattern of dropouts during calls

We've had a Unity-based VR app using Photon Voice working well for some time. We're currently on PV v2.31.

Just in the last couple of weeks we've had a troublesome emergence of some weird behaviour. What we're observing is that when we have a room with 3+ users, sometimes a user's voice will simultaneously drop out and not be able to be heard by other users. Our testing shows that on the dropped users's system, Photon Voice is still reading from the microphone and delivering samples (we have a level meter which shows this; it is attached as a post-processor to Recorder, and we read the samples in IProcessor.Process). We have added the VoiceDebugScript too, and when a remote user calls CantHearYou on the dropped user, it reports "no idea why!, should be working". We have observed dropped audio from users on Android and Windows. We don't see any odd messages in the Unity logs.

Two things seem to restore the user to the audio mix; if any user leaves and rejoins the room, or sometimes it just comes back after several minutes.

One clue we have is that it seems to affect users on higher-latency links more often. E.g. if we host the audio room in US West, the affected users might be in Australia or Spain, and users in the US don't drop their audio.

At this stage, we're out of ideas and running out of additional things to try, especially as it is a hard bug to reproduce on-demand... it seems to depend on the particular mix of users in the room and the random conditions at the time.

Since it appears to us that the user is dropped at the Photon server end, not at the originating user end or the receiving user end, and the recent emergence of this problem, it leads us to ask if anything has changed recently in the way Photon Voice's servers are handling incoming audio streams? Considering the possible issue of latency, if Photon's servers are dropping incoming audio streams that have higher latency than the average across all users, is there some parameter we can set to avoid that?

We have a build updated to PV v2.50 which is shortly to begin testing, so that is something we can update.

Comments

  • We have updated to PV v2.50 and are observing the same problem.

    Our code base has been stable. We first observed the problem on Oct 18th, although it could possibly have been occurring a few days earlier.

  • Hello, my users are also experiencing this exact issue (Unity, PUN2, PV v2.24.1) which has started mid-October. Prior to this for the past year, we've had extremely reliable voice communications with PV.

    Users who lose voice comms are reporting that others can only hear them after activating our test microphone feature (debug echo mode).

    Please let us know if something has changed server side and we need to update a setting within our app.

  • We're seeing the exact same thing. Happens on Quest and PCVR so doesn't seem to be local or system related. We actually saw the first instance of this on Sept. 30, but it's gotten much more prevalent. We've gotten android system logs which don't show anything unusual, and have been trying to capture full Photon Voice logs but very hard to reproduce when we need it. Always in a room with 3+ people, and we have a setting in-game to toggle the mic off and back on, which always fixes it for a minute or two before it often happens again. It's definitely more common on systems with less stable internet connections, as I've personally had it happen a few times when I was connected to my wifi range extender.

  • We are also experiencing the same issue (Unity, PUN2, PV v2.23.1) which has started October. Happens on HTC VIVE and HTC Focus 3.

  • justinl604
    justinl604 ✭✭
    edited November 2022

    We're also experiencing this exact issue on Quest. The issue occurs both in PUN and Fusion. Started early October. Flipping mic off/on also fixes it but sometimes just for 30 seconds. Sometimes if one of the users in the room leaves, it resolves things for everyone. But not all the time.

    We also noticed that it happens more commonly on systems with less stable internet as it more frequently happens to one of our developers who admittedly has poor router issues in his house.

  • We asked on a dev discord, and 4 additional published games all said they have been trying to figure out the same issue. All started at the same time, mid-October but we've got one report from as early Sept. 30. I can reliably get it to happen by using my dodgy wifi range extender. If I switch over to the main access point there's no mic issues. Seems like some sort of packet loss causes voice to disconnect but PUN keeps working? We even got a log of it happening in editor with full Photon Voice logging on and nothing stood out, which makes me think it's server-side.

    I was about to blame Oculus, but multiple confirmed reports that it's happening on SteamVR with Vive, G2, everything. If there's anything more we can do to help get to the bottom of this, just ask. Our support team is getting underwater on this, and Black Friday will open the flood gates.

  • Thanks everyone for chiming in and reporting the issues.

    I am sorry we didn't update this thread from our end but we got the reports via Discord, aggregated the info and looked into the issue.

    Today, we took several new Photon instances offline, which likely caused the issues. We were running variations of the server, which were tested positively before but apparently failed in combination with live Voice clients.

    We think this pattern of issues should no longer happen. If it does, please report (in best case via mail including the AppId: developer@photonengine.com) and we look into it.

    Sorry for the hassle and thanks again for everyone providing useful insight!