The first time a BehaviourTreeOwner is enabled or in it’s Awake (whichever is called first), the assigned BehaviourTree is getting initialized, which basicaly is deserializing from JSON. This is something that is somewhat costly and the reason you might experience this.
Are the pooled agents loaded during the loading screen have the chance to get their Awake called? If so there should be no performance hit when you enable their behaviour later on.
Are you maybe instantiating the agents disabled?