AllViaServer small delay

Options
Hello guys ! I'm currently working on a multiplayer version of a game. I already have the room joining/listing working properly, and I'm having some trouble with the game mechanics.
In our game, the player must pickup items which he can upgrade in furnaces or workstations. During local play, no bugs happen. Only one item can be put inside a furnace and only one player can work on an item in a workstation at a time.
But in the on-line version, when two players used the same workstation at the same time, a glitch happened in which they would both use it and get stuck. To fix this, I changed almost all the object interactions to use the AllViaServer RPC option (they were using the All option before). This fixed the bug, however, there is a small delay (about 300ms) between the button being pressed and the interaction happening.
I imagined that, since the action travels to the server first, it would generate a small delay. But is there a way to reduce it substantially?
Other than sending the RPC to the master client (in which there would also be a delay), I see no other way of fixing this.

edit:
Player goes to a storage (which has a photon view) > player presses the button > storage instantiates an item using photon > item interacts with the player using an rpc (using all option) > player is now carrying the item

Player goes to the desired workstation > player presses the button to interact > the workstation sends an rpc to upgrade the item (using AllViaServer) > if the workstation is not being used, the player starts upgrading the item

Thanks for your time !