FPS drop when deactivating gameobjects where SetAnimator has been called

I get a rather big drop in frame rate if I disable gameobjects where I have activated SetAnimator(myAnim).

Scenario:
When the scene is loaded I instantiate 10 BoltPrefabs and, in their Attached() function, SetAnimator is called and then the gameobject is disabled. My fps drops from 140 to less than 10 if I run in the Unity Editor. If I remove the SetAnimtor line everything is fine.

I activate / deactivate these object as they are pooled.

Is this the intended behaviour? is there a work around?

For the moment I have this workaround that at least makes it playable but I am hoping there is a better way:

private void OnEnable()
{
if(GameManager.Instance.IsNetworkServer())
{
Debug.Log("I activated the animator!!");

state.SetAnimator(animator);
state.Animator.enabled = true;
}
}

private void OnDisable()
{
if (GameManager.Instance.IsNetworkServer())
{
Debug.Log("I TURNED IT OFF!!");

if (state.Animator != null)
state.Animator.enabled = false;

state.SetAnimator(null);
}
}

Cheers