So, I have an idea, but it would require a fair bit of work. I kind of love it, though.
Also, I spoilered the wall.
Spoiler
Pre-battle
When you bump into another fleet you are given the choice to ignore or pursue. If both parties choose ignore, combat is avoided. If either side chooses to pursue the other fleet, we head to the Deployment Phase.
Deployment Phase
All ships in both fleets start out deployed. So why call it the deployment phase? Well, you know all those objectives on the map? You and your opponent can deploy them (they would also work a bit differently, explained later). Also, during this time you can choose the individual starting positions of your ships (within a certain radius of your starting position) and where to place any System Drive Interdiction Field Generators (placeholder name, explained later, hereafter referred to as "Interdictors"), if any.
The battlefield would be borderless and the default state would be that you can engage your system drive anywhere on the field. The way the system drive would work is that you press a button- I'm thinking 'C'- to increase your speed beyond its max, and once it reaches a certain speed it leaves the battlefield (when a player hits that speed, they could get the dialog box asking if they mean to retreat). Releasing 'C' maintains your current speed, and you slow down by pressing 'S' (or 'W' if you want to slow down to your top speed). You might not want your opponent to do this (and viceversa), so that's where Interdictors come in. I'll talk more about them later, but they prevent use of the system drive within their area of effect. Also, this is a good time to explain how deployment would work. EDIT: Now that I think about it, activating system drives might not even require its own special key. Just make 'W' accelerate without limits when outside of interdictor bubbles, release 'W' when you're happy with the speed you're going.
So, both fleets start at different spots on the map, relative to their positions on the campaign screen (ie. bigger fleets start farther apart, relative positions depends on your angle of approach- you might start at the 2 or 9 o'clock position, for example EDIT: not necessary, now that I think about it). You know the area where the enemy fleet can deploy, but nothing beyond that (because fog of war). You can deploy anything (besides ships), anywhere on the map, but you'll only want to deploy them where they'd be useful. The beacons we already have work basically the same, but are instead deployed by the participants (and start out 'captured' by the deployer), they have a limited area of effect, and the bonuses don't stack if they modify the same attribute. You wouldn't want to deploy them far away where they're useless, but you wouldn't deploy them too close to the enemy because it makes them easy to capture.
Deployables can be captured (except interdictors affect friend and foe alike), but they can also be destroyed. Since your enemy can flee at any time, you may want to deploy an interdictor near them to stop them from retreating. If your enemy is focused on retreating, then at this point they have two options- get out of the interdiction bubble and engage system drives, or destroy the generator and engage system drives. Depending on where you place it, you can have their system drives be jammed and yours unjammed, but it might leave the interdictor vulnerable. (I'll talk more about deployables later, but this is just to give an idea of the tactical and strategic considerations this allows for)
Combat Phase
Combat would behave much the same way it does now. Inside the effects of interdiction, the meat of the combat is essentially the same. You can even explain the Burn Drive as being "hardened against the effects of interdiction, allowing the system drive to be engaged for short intervals under jamming." Most combat will be centered around the deployables, for the bonuses they provide and some other reasons to be explained later, but it doesn't preclude the possibility of combat outside the bubbles.
Figuring out how to handle ships entering an interdiction field is a bit tricky. They could be going extremely fast thanks to their system drive, but that could be unbalancing if they can be that fast inside the bubbles. I feel like they should quickly decelerate to some upper bound for their ship size.
The framework allows the possibility of mid-combat deployment of beacons and bubbles as well (mid-combat deployment may require a specific type of ship). You could even have a ship system that allows it to be a mobile interdictor.
Post-Combat Phase
Most of the post-combat stuff remains the same, with some differences. If you win (ie. you have at least one ship on the battlefield while your opponent has none), all deployables you have captured at the end of battle become yours. Anything you don't have captured has a chance to be sabotaged (retreating/destroyed fleet sends self-destruct signal to the deployables, you attempt to jam, could be affected by skills/hullmods). If you lose (ie. you retreat, get destroyed), you lose every deployable you deployed because you abandoned them.
There will also be no need for RNG causing ships to be disabled or destroyed post-combat because there's no need to simulate post-battle pursuit; the ships have already retreated out of range with their system drives.
Deployables
This comprises all the things you can deploy on the battlefield. You have your beacons that buff you (lots of possibilities for that) and interdictors that inhibit the use of system drives, but there's a great deal of variation you can have with that. The deployables could have greater area of effect, provide larger bonuses, be more durable, take longer to capture, or even put various debuffs on the enemy depending how much you're willing to spend (and risk when deploying them). You would have to buy or capture these if you want to deploy them, so there could be a great deal of variation in their cost and size. You could even have cheap Sensor beacons that simply give you lots of visibility, which could be important on a potentially large battlefield.
Also, just brainstorming, there could be some interaction between deployables and phase cloak invisibility. Detecting cloaked phase ships in combat could require it to be within range of one of these sensor beacons. If the deployable is cheap and light, there's no reason not to have at least one, especially with fog of war being an issue. You'll really want to defend your sensor beacons if it means you won't be able to detect phase ships without them.
To explain how things are deployed pre-battle, I imagine they're launched out just before the battle begins, they drift into position, and they stop by using maneuvering thrusters or some such, just in time for when the battle begins. It wouldn't really require a specialized ship. For mid-combat deployment, I imagine that would take specialized ships to get into position and deploy them- let's call them "Construction Rigs".
Construction Rigs would essentially be used for mid-combat adjustment of the 'terrain.' They could also be used to repair nearby friendlies if that's a possibility. Not only would they be able to fly around putting up beacons or additional interdictor bubbles, they could also un-deploy them. Maybe you decide to retreat and don't want to leave that expensive, top-tier beacon behind. Maybe you just want to pick it up and move it closer to the battle.
It should take some time to un-deploy (maybe dependent on how long it would take to capture it and how large it is). Larger deployables could require larger Construction Rigs to deploy and un-deploy mid-combat. Not sure they should be able to un-deploy interdictor bubbles mid-battle, though. At the very least, it should be a lengthy process to do so.
I like this because it explain where the hell those beacons come from in the first place while at the same time giving the player and AI the ability to shape the battlefield in interesting, meaningful ways.
EDIT: Oh, you could have some deployables be armed, even make them dedicated weapons platforms, you could set up minefields... lots of possibilities.
EDIT: Decided to flesh out my idea of deployables a bit more.
So, you could have deployables be distinct items that take up inventory space and have all their stats predetermined, or you could make them modular. You could even make them take up FP and hangar space instead of cargo space (basically, treat them like crew-less ships that can't move and can be captured mid-combat). If they're modular, they could come in small, medium, or large varieties with Main System Beacon slots for different buffs (like Sensor, Navigation, or even Interdiction). You could even have different 'hulls' within their size class that have weapon slots or maybe an extra Main System Beacon slot.
Just some stuff to think about.
System Drive Interdiction
There could be a few different types of interdictors to allow for more varied battlefields. The one I refer to throughout this post would inhibit the use of all system drive activation within a certain radius (a bubble, basically). You could also have a 'ring' type that has a wide area around it that is completely unaffected, but a small-ish ring around it far away where the interdiction occurs.
Imagine an archery target- the bullseye would be like the interdictor bubble and the outermost red ring would be like the ring interdictor. You could also have weaker interdiction that affects a larger area, but allows some use of the system drive. All are interesting ways to shape the battlefield.
As for mobile interdictors (ie. interdictor as a ship system), you could have any or all of the above, but you could additionally have a single-target interdictor.
EDIT: You could also (as a default that appears of its own accord without any deployable) make a large-ish, circular interdiction field between the two fleets that may or may not encompass both fleets' deployment zones to create a battlefield, even when neither fleet has a deployable. Could be explained away as the interaction of the fields surrounding both fleets as they clash distorts space-time and disables the use of system drives. In smaller engagements, it could easily encompass both deployment zones, but since bigger fleets start farther apart, it could necessitate the use of deployable interdictors to prevent the enemy fleet from immediately fleeing.
I tried to flesh out the idea as much as possible, but feel free to do whatever you want with this.
EDIT: I just realized I never actually said exactly what interdictors do outside of System Drive Interdiction. Derp. Fixed that and put System Drive Interdiction back where I prefer it.
1) Remove borders, allow use of system drives in combat. Create 'soft' borders through mechanics that disable use of the system drive.
2) Create a new class of 'ship' called deployables (placeholder name) which act much the same way the current objectives on the map do, but re-use much of the code already in place for ships. The player and enemy AI would be the ones deploying them.
3) Create a Deployment phase where you have time to set up the positions of your fleet and deployables. Allows you to create formations and influence the 'terrain' of the battlefield.
There's also the work integrating it into the current system (relatively minor, compared to actually implementing the features), balancing everything (relatively major), and teaching the AI how to handle these things (relatively major).
