By back-off do you mean a move order or a retreat order? If the former it happens all the time, it's like a low priority for the AI and will mostly ignore it, which is why it is better to simply order it to retreat as the behaviour is immediately changed to "avoid enemies, then leave"
If you want your ships to survive it's better to do retreat then cancel the order than asking it to move, especially against a station were they will never listen to positioning once at firing range (tried to focus on a side so many times, only to have the warships alone all around the station, and the carriers on vacation, so just attack order everything and out-dps the thing)
Funny thing is that's not entirely accurate either. Just today I had a couple missile carriers take position well out of a station's range while I went in with my flagship and, a few maneuvers and a little drifting aside, they mostly stayed put and fired missile support like I intended. Neither of them were Prometheus II's this time of course, so no burn drives...I also try not to use "retreat, then cancel" tactics because if I use the "retreat while defending" they never get anywhere if they're already under fire and if I use "direct retreat" they tend to turn their engines to the enemy and...you can imagine how that goes.
I'm all too familiar with how senseless AI gets when engaging a station with no orders--fight allied ships for position, close in, bang into each other trying to get a shot off, then blow up in a chain reaction when the station finally pops in their face--but that problem seems to be more about them having a clean shot be too high a priority or something. This is a far more specific problem and I feel like it's directly tied to the way the AI uses active systems with abandon even when doing so is an absolutely terrible idea AND rather contrary to their orders.
I don't know how helpful it might be but I have to ask Alex directly here: Have you ever played Gratuitous Space Battles? The entirety of player control happens during ship design, fleet placement and pre-battle orders; this game sometimes has a similar feel in battles (which I like.) I wonder if there's inspiration to be taken from the way it handles fleet orders? Lots of priority sliders, drag-and-drop escorts and formations, lots of emphasis on giving smart orders to account for circumstance, that sort of thing. It's much more complex than I think you want this game to be but I wonder if perhaps a little methodology might be applicable here?
