Currently, enemy fleets will only ever retreat when the majority of their forces are wiped - essentially a rout.
I think it'd be not only more believable but also more forgiving at times, if NPC fleets were capable of recognising when a battle is in danger of being too costly for them.
Say for example you're in a small fleet with a couple destroyers and a few frigates, and get caught by a large fleet. You manage to take out the majority of their frigates, a few destroyers and start eating into a capital or two. At this point, not all fleet captains and not all ship crews would realistically be willing to continue, even if they ultimately maintained an advantage.
Depending on the dispositions of both fleets, they may retreat off-map and re-deploy to regain initiative, or simply decide to cut their losses and fully disengage - giving the player the option of fighting a pursuit battle against them, and the chance to recover ships lost in a defensive fight.
This could be influenced by faction, the fleet's objective (ie. why are they fighting you), officers/personality/experience, whether you're in a system they have a market in or whether you're out in the middle of nowhere; the rapidity of enemy ships' destruction and their crew complements, remaining ships' CR - and more, probably.
In other words, players would more often be able to come out on top as the underdog in a given battle, depending on its context. It would result in situations where you might have taken heavy losses of your own but your fleet ultimately survived and recovered some of those casualties, without having necessarily won the battle decisively, or chosen to flee and fight a disengagement battle.
Depending on the campaign situation, it might only amount to breathing room or it could be the chance to slip away - but the aim is to help reduce the incidence of situations where the player is so heavily outgunned that they can't "clean"-disengage without major losses, regardless of how much effort they put in.
I guess it's also another way of loosely simulating not only fleet-wide morale, but also the enemy's logistical and strategic concerns - to them, after losing so many ships, is wiping your fleet worth how many more you'll take with you?