@Tei and @Torch: were either of you using the mods Disassemble Reassemble or Arsenal Expansion?
I've just been discussing a very similar bug with Surge on the Discord, where all ships of a side appear to have retreated yet the battle doesn't end. He pointed to the Vagrant-class Destroyer from Disassemble Reassemble. Specifically, it seems to be caused by ships with separate modules (the Vagrant has a modular chunk of armour) - InventorRaccoon mentioned to me that a similar issue again was reported with Arsenal Expansion's Flourish, a high-tech Lasher analogue with a module that creates a secondary shield.
I tested a mission myself, two identical fights except one involved Vagrants and one didn't. Both times I deployed all forces then immediately ordered a Full Retreat. Without the Vagrants, the battle ended after a couple of seconds of the last ship retreating. With them, it didn't, it just stuck on an empty battlefield for about a minute before I exited the mission manually. The armour modules never showed up separately on the tactical map, so it's hard to tell if they were "stuck" off-screen. Regardless, it's looking pretty clear that this is related to ships with modules.
Is it possible that modules on ships aren't being removed from the combat engine when the parent ship retreats? Even though they don't appear on the tactical map, does the game still think they're in play and not end combat because of that?