Player Leaving Room Destroys all Player Objects for other Players in Room

Options

I'm having an issue with my project where I'm trying to integrate Fusion into an XR Interaction Toolkit demo project. I spawn the player's rig when they connect to the server and all player rigs in a room seem to work okay right now, but when a player leaves all the player rigs for everyone still in the room get destroyed besides the rig for the player that just left. I already tried integrating XRI into the Fusion Dragonhunters sample but the rigs were broken. Has anyone else faced issues like this?