To stop the game from going outside its memory allocation, set a global number of maximum Fleets that's tested and known to be safe as an upper limit.
If people want to mod it to go higher, then they should change the global (it'd be nice to have a memory footprint, too, so that messing about with limits is better-understood).
There should be some boolean in the fleet-creation operation that, if invoked, replaces the smallest Fleet (by CR rating or whatever variable(s) are now used in place of FP) that is not near the Player Fleet (if possible) so that forced situations can operate properly regardless of limits (i.e., if a script needs to generate a Boss Fleet or three, let it be possible rather than fail).