Rigidbody Erratic behavior on instantiation until input synchronization

Options
Basically working on a 1 on 1 racing game. Using the PhotonNetwork.Instantiate using a PhotonTransformView object on the vehicles.

Problem: When the networked (not the master client) is spawned it is erratic until the opponent uses the vehicle controls and then it receives the synchronization changes and works as expected. I am at a loss on why when it first gets instantiated it goes forwards, backwards, bounces, and may stop against the barrier wall and as soon as the other player hits the throttle or some other input it snaps back to the starting line and takes off like normal with nice smoothing and everything? I was thinking maybe there was a way to force the position until the race started and then let synchronization take over but not sure how to approach that and not interfere with the photon functions.

I am still learning a lot and my first real time doing multiplayer so any help is appreciated.