Currently, Vengeful relations occurs when reputation falls below -75. The consequences mean the faction hates you forever, with no way (in unmodded game) to regain standing. It also blocks access to all markets. Even smuggling is not an option anymore.
What causes reputation to fall below -75? Currently, if you pursue the enemy while your reputation is below -70. The -5 reputation afterwards in not bound by the usual -75 cap for normal fighting.
Who would you pursue? Badly damaged warships from a previous round? All freighters in a relief fleet because the enemy never deploys civilians in combat, unless forced by pursuit?
Because Vengeful cannot be recovered from, points below -76 are meaningless - the effect between -76 and -100 is the same.
Assuming the consequences of Vengeful relations remain as is, that is point-of-no-return and no trading allowed, what should cause it?
Killing fleeing ships should not, AI of enemy factions does it all the time against each other, and we never see relations at Vengeful with each other.
When fighting (at least in other RPGs), it is a good idea to take no prisoners because survivors may know too much, and the only way to prevent that is to kill everyone. Also, leaving survivors is a loot and XP tax. There is a reason why adventures are called murderhobos in classic RPGs.
Vengeful should be earned when you do something so outrageous and big enough to be a major strategic blow to the faction, enough to redraw the map, enough to be a major turning event during the game. Killing survivors from a fight, or even intercepting relief fleets bound to an enemy should not result in Vengeful; it is war after all, and those acts are probably limited to tactical in scope. Something strategic, like capturing (or destroying) a major base or homeworld, should earn the enmity of the faction attacked. For example, Jangala looks like a major Hegemony world. If I can invade and capture it (then use its resources - hello blueprints and autofactory! - against Hegemony), that should enrage the Hegemony so much that they want me dead above all else.