The Suggestion: Let players who have fleets that are too big to retreat choose ships to sacrifice so the rest can attempt to escape. If the players fleet is sufficiently larger than the enemy fleet (double the fleet points in combat ships?) let them disengage with the harass effect, or make them select a few ships to send to their doom.
You can say you’re sending a decoy fleet to give the rest an opening, or something to that effect, and if you have the larger fleet say something to the effect of “marshaling so many ships in an escape effort will take more time than you have. You’ll have to send a force to ensure you can escape.”
The logic: Not being able to disengage because your fleet is too big seems a bit silly, but being so badly surrounded that there are no good escape angles makes a lot of sense.
I understand the reason players can’t run for it when the fleet is too big now is because it would be used to cheese a whole massive fleet onto the battle space. Makes sense, but having to intentionally suicide ships the player doesn’t want to save seems like a waste of time, and just makes the battle painful to play out.
This option will make choosing to run with a large fleet a loss, whether the player cheeses the encounter or not. Choosing ships to sacrifice stings, but it at least lets the player skip the infuriating experience of trying to get their fleet to die in an unwanted combat, while still punishing players trying to abuse the system.