This mostly seems to occur in (relatively) 1v1 situations, such as capturing locations.
Not a bug, its a feature.
when ordered to capture a location ur ship draws an invisible circle around the location, starts putting invisible "engage" orders on enemies inside it that only it (& other ships assigned to the same location) can see & ignores enemies outside that circle. When it beats an enemy ship but it manages to back out of that circle it lets them go & goes back to cap the location. This is the only way it makes sense for it to work for 2 reasons
1) if the only thing the developer knows about a 1v1 over an objective is that the ship overloaded manages to back out of the contested circle then it can be inferred that that ship is faster than its opponent & chasing it won't do much good.
2) when you assign the ship to capture the location, the ship operates under the assumption that you want it to capture that location. What other assumption can it make? & letting enemy ships go instead of letting defeated enemies peel them away from the location leads to it capping that location & no other action will.
Like, in Heroes of the Storm theres an extremely weak hero who can drop an egg on the map & when u kill him it A) doesnt give u anything and 2) he respawns at his egg instantly. So in order to bring value to his team, his job is not to run up & try to push the enemy team off of the objective, his job is to drop an egg near the objective then run up & annoy the enemy team in an attempt to get them to chase him and then lead them around the map until they kill him then get respawned right next to the objective & cap it before the enemy team can get back to it. Dragging the enemy team off of the point to chase him for a pointless kill is
effectively the same as beating them in a fight & if he's even able to distract 2 of the enemy then his team has a game-changing advantage. So when I see my team getting peeled by him I have to jump into chat & be like "stop chasing him & go cap". And that's almost the exact same thing that's happening in starsector, except that your team
already knows not to get peeled. It is simulation the experience of having a smart & effective team in this specific situation.
If you want them to press their advantage & kill the ship they're engaging then you need to switch them to an eliminate order on the ship that's contesting the point, as this will automatically assign another ship to come cap while he's off chasing. It's a bit inconvenient to have to be aware of when ur ships are 1v1'ing but what can you do, the game is better for you having to do that