if you're already hostile to that fleet, i don't think reputation hits would be enough to incentivize against shooting up your 'allies', in most cases.
if all three factions are involved in the ensuing battle, i'd prefer them temporarily allying against the player, simply because it's the scenario that creates the most challenging battle. if the player fleet is more powerful than the other two combined, fighting only one of them is likely gonna be rather boring. having the other side as allies would make that even worse.
if the player doesn't command a fleet of overwhelming power, they can either wait for the AI battle to end, then pick of whoever is left. or they can risk going against both enemies at once. the later is faster, and could be used to keep other fleets from stealing your bounties, but comes at the cost of a potentially much tougher fight.
that way, there also doesn't need to be an arbitrary rule like "you can only break up hostile battles if your fleet is this strong, relative to theirs". rather, you can always choose that option, but whether it's a good idea to do so simply depends on your ability (through fleet power + player skill) to come out on top of the ensuing battle, which i think would make it a much more interesting option.
although it might need some kind of warning-popup to make sure the player understands what they're about to do, so they don't accidentally choose to fight a battle against a much more powerful enemy force than they intended.