I realize this is a huge necro, but:
Here's an idea.
1. There should be capturable points that influence CR decay, and points that gradually reload Missiles. This would hugely encourage players to not kite battlefield edges and use a fleet.
2. Retreating should not reload Missiles, ever, but it should allow the ship to return to the battlefield later. I.E., retreating a badly-damaged Cruiser just puts it back into the Deployable list; it doesn't take it off the playing field, for the player or the AI.
3. Retreating or entering the battlespace should involve turning on the Travel Drive. This means ultra-fast movement (think a blurring motion until it cuts off). Travel Drives should take between 15 and 45 seconds to warm up; hitting Retreat on a ship means surviving for a while and then escaping rapidly if you survive that long.
4. The battlespace should be essentially infinite, but the AI should not stray outside a zone encompassing the Objectives, unless Retreating, other than Fighters.
5. Player ships taken out of this zone should be allowed to do so, but if the AI captures the points that drain CR, this won't work out.
6. Ships that have reached zero CR outside the zone should be Disabled, further penalizing attempting to escape the AI.
7. Because ships can be brought back into a battle, but Missiles won't be reloaded if the entire fleet Retreats, there aren't any good reasons to leave in the middle of a battle you can win, nor, with the other changes, any good reasons to loiter outside the AI zone. Your own AI ships won't leave the zone, either.
So, that would solve most of these problems. I don't have any issue with the player sending a damaged ship out of the fight and entering a new one to fight. The issue here is that corner-kiting, wall-hugging, abusing the Retreat system to reload Missiles or recover CR, etc., etc., are all boring and reward cheesing rather than fighting. Merely making sure that a player who refuses to fight will lose their ship to CR degradation will keep players in the zone; if players can get Missile reloads by holding points, then Missile-heavy fleets are viable without resorting to an abuse of the game-mechanics. And the Travel Drive and a large circular zone means no more "Oh, I just killed the first wave, and now the second wave magically enters the fight and crashes into my fleet that's chased them to the edge of the battlespace"- ships should arrive in groups, but groups scattered around a 90-degree arc of the zone, so that that artificial issue quits influencing gameplay.
I think it's still possible to test this stuff out, but meh, I have no idea whether my custom-battlefield code from Vacuum is still even vaguely serviceable, so it's probably not trivial to set up a test for these ideas. I think this solves most of the current problems, though; it gets rid of the invisible-wall issues pretty much for good, makes enemy reinforcement waves arrive in unpredictable places, quits rewarding players for using Retreat All by magically reloading their weapons, etc., which are all things I think we can agree aren't great for gameplay.
...so, I built something like this. It'll be in the next version of the AI, if I don't port it out to another sub-mod.
Basic rules:
1. In Missions and the Campaign Missions, in non-Escape scenarios, incoming ships spawn on two edges of the battle-space, which is now a circular area, instead of a rectangle.
2. When spawning in, the ships use their Travel Drives.
3. Attempting to leave the area when not under Retreat orders triggers the ship turning towards the center of the battlefield and turning on the Travel Drive until back within the area.
4. When the ship has Retreat orders, it swiftly turns and then engages the Travel Drive, leaving rapidly.
The resulting combat dynamics are actually quite good. This system:
A. Prevents using corner-cheese / edge-cheese tactics entirely.
B. Actively punishes players attempting to ride the battlefield boundaries, and prevents dumb AI-vs-AI chase scenarios (you know, that boring stuff where they chase each other down the walls).
C. Makes the initial deployment a lot more chaotic and interesting (I'll probably do some tweaking on this to make it feel a bit more like a formation flying in, with Frigates out front, etc., as I refine this).
D. Makes the rush to objectives much more random, instead of a giant predictable blob.
E. Makes scenarios where the enemy gets multiple waves much less favorable to players who've pushed up the battlefield- you cannot squish the next wave against the walls any more and the position of incoming forces is less predictable. And enemy forces no longer kill themselves running into the wreckage of dead ships nearly as often.
In short, it worked as proposed; the prototype's still rough, but it already feels better. I think it solves a lot of the problems we talk about with this area of the game design.