Here's an idea.
1. There should be capturable points that influence CR decay, and points that gradually reload Missiles. This would hugely encourage players to not kite battlefield edges and use a fleet.
2. Retreating should not reload Missiles, ever, but it should allow the ship to return to the battlefield later. I.E., retreating a badly-damaged Cruiser just puts it back into the Deployable list; it doesn't take it off the playing field, for the player or the AI.
3. Retreating or entering the battlespace should involve turning on the Travel Drive. This means ultra-fast movement (think a blurring motion until it cuts off). Travel Drives should take between 15 and 45 seconds to warm up; hitting Retreat on a ship means surviving for a while and then escaping rapidly if you survive that long.
4. The battlespace should be essentially infinite, but the AI should not stray outside a zone encompassing the Objectives, unless Retreating, other than Fighters.
5. Player ships taken out of this zone should be allowed to do so, but if the AI captures the points that drain CR, this won't work out.
6. Ships that have reached zero CR outside the zone should be Disabled, further penalizing attempting to escape the AI.
7. Because ships can be brought back into a battle, but Missiles won't be reloaded if the entire fleet Retreats, there aren't any good reasons to leave in the middle of a battle you can win, nor, with the other changes, any good reasons to loiter outside the AI zone. Your own AI ships won't leave the zone, either.
So, that would solve most of these problems. I don't have any issue with the player sending a damaged ship out of the fight and entering a new one to fight. The issue here is that corner-kiting, wall-hugging, abusing the Retreat system to reload Missiles or recover CR, etc., etc., are all boring and reward cheesing rather than fighting. Merely making sure that a player who refuses to fight will lose their ship to CR degradation will keep players in the zone; if players can get Missile reloads by holding points, then Missile-heavy fleets are viable without resorting to an abuse of the game-mechanics. And the Travel Drive and a large circular zone means no more "Oh, I just killed the first wave, and now the second wave magically enters the fight and crashes into my fleet that's chased them to the edge of the battlespace"- ships should arrive in groups, but groups scattered around a 90-degree arc of the zone, so that that artificial issue quits influencing gameplay.
I think it's still possible to test this stuff out, but meh, I have no idea whether my custom-battlefield code from Vacuum is still even vaguely serviceable, so it's probably not trivial to set up a test for these ideas. I think this solves most of the current problems, though; it gets rid of the invisible-wall issues pretty much for good, makes enemy reinforcement waves arrive in unpredictable places, quits rewarding players for using Retreat All by magically reloading their weapons, etc., which are all things I think we can agree aren't great for gameplay.