How to create a private Audio group?

Hey guys, I am creating a meeting table for people who join the table, they will have a private audio group which means only the people on the table and talk and hear each other (Remove the initial audio group), here's my script:

At Awake() method, I move everyone to a general audio group which is "1";

At AudioGroup2() method, I create a function to move all the people on the table to private audio group 2 and remove their speaker to play voice message from audio group 1;

At GeneralAudioGroup method, this is for people who're leaving the table which is the private meeting audio group, so I just return everything back to the initial mode.

But the problem is - When I sit down on the table, I still can hear the player's voice outside from the table which means the method didn't remove the voice message from audio group 1, do you guys have any idea? Looking forward to your reply, thanks in advance!