I'm never going to properly finish this suggestion so I'm dumping it here as-is for inspiration/idea mining/whatever you want to do with it. Gah, it's frickin' long - and better formatted than I remember...
Somebody was complaining about every faction and their dog in the modiverse picking on the Hegemony a while back. A couple people replied with several valid reasons why it is the way it is, but I want to add another: no relationship besides Hostile means anything between factions.
The relationship between Tri-Tachyon and the Hegemony is a prime example of the same problem in vanilla. In-game they are in a shooting war, but look at all the lore and it's clear they're not really meant to be actively fighting.
I feel it's easier to describe the negative relation levels in practical effects, so I'll only try my hand at them.
To start off,
Suspicious.
Spoiler
Suspicious factions don't like each other. They're often just looking for an excuse to start a fight. They stop each other's trade fleets to "look for contraband", but really they're just harrassing them. They'll even go a step farther than that by hiring deniable mercenary raiders.
Mercenary raiders have a somewhat complicated behavior pattern. First, they spawn as an "independent mercenary patrol" (or another faction neutral to both factions) at a market belonging to their employer. Then they cruise to where they want to raid, move away from any fleets, and go transponder-off, still pretending to be a merc patrol. They proceed to attack trade fleets belonging to their employer's enemies. If any transponder-on fleet survives a battle with them, their true faction and fleet name is revealed. They continue to raid for a while or until they lose too many ships, then they move away from any fleets, flip on their transponder, and become a faction mercenary patrol (if they aren't still luckily disguised as independents). They, of course, then do the standard return, stand-down, despawn procedure.
Faction "privateer", but not "bounty hunter", fleets come with the mercenary territory.
Patrols counter these raiders by intercepting transponder-off fleets, demanding to know who they are, and then demanding they turn their transponder on if their faction cares about that (or attacking/running away if the fleet is from an enemy faction). Disguised raiders can bluff their way past a transponder check once, but the second time they lose their disguise. Another factor working against raiders is that friendly fleets won't accept help from transponder-off fleets, just like with the player.
For the player, disguised fleets have two special, but related mechanics. The first is that attacking them while they are transponder-off only has a small impact with their faction. The second, more important mechanic is that attacking them while they're disguised (unless the player is transponder-off) has no effect on independent reputation; if they are transponder-on, however, it is like attacking a transponder-on fleet of their faction. Also, if the player sees them doing raider-y things while disguised, like attacking a fleet, then their true name is revealed to the player (but not their faction).
is much briefer, I promise.
Spoiler
Inhospitable factions refuse to trade with one another. They also supplement their mercenaries with unofficial raiders drawn from their military. Faction raiders act exactly like mercenary raiders. On the homefront they replace some of their trade fleets with disguised counter-raider patrols.
There is one dramatic difference from Suspicious: Inhospitable factions will attack each other's transponder-on fleets within a certain distance of their markets. They don't give chase outside of that radius, though.
factions act pretty much exactly like now.
Spoiler
One difference is how their raiders act. They'll drop their disguise as soon as they go transponder-off and they'll turn their transponder on right before they attack (allowing them to support each other). Additionally, when they return home to stand down they'll keep their transponder off until they are close to home; only turning it on if they are attacked.
This would also be a nice opportunity to add blockades. Blockades are composed of one or two big fleets and a bunch of picket fleets that go and surround a Hostile market. The blockaded faction stops spawning fleets from that market, and instead spawns blockade runner fleets at its other markets. Blockade runners are like (heavily armed) trade fleets, except they immediately despawn upon reaching their destination. I imagine a mighty system defense fleet would eventually be formed to break the blockade.
The PK devices come out when two factions are
Vengeful.
Spoiler
I doubt planet killers, or anything else that wouldn't show up at Hostile, will ever actually be added to the game, but if they were they would go here.
Despite all of the above, I was really thinking about pirate raiders when I started all this, so let's talk pirates (and independents).
Pirates:
Spoiler
Pirates have a special behavior different from other factions: they'll fight together even while transponder-off, but only near a pirate market.
Another unique feature is you get to see the true names of disguised pirate fleets if you are on good terms with them.
Pirate Raiders act very similar to faction raiders; the main difference is they want to extort people rather than blow them up. Unlike faction raiders, pirates can spawn disguised or not. Undisguised pirates spawn at a pirate market and disguise themselves (faction only if the player has gotten close enough to see their name) after leaving its proximity. If they're still somehow disguised when they decide to RTB, they'll turn their transponder on and go home (dropping it once they're close if home is a pirate market). Otherwise they make a beeline towards the nearest pirate market.
Smugglers also disguise themselves, but as mercantile convoys. Like pirate raiders, if they spawn at a pirate market they disguise themselves only after leaving. If they're not coming from or going to a pirate market then about the only way to tell they are smugglers is to notice that pirate raiders aren't bothering them. They will run for the nearest pirate market if they lose their disguise.
Independents:
Spoiler
Independents turn their transponder off when they are in hyperspace, unlike faction fleets. Fleets disguised as independents mimic this behavior.
Idea: Some independent fleets such as patrols, mercantile convoys, and privateers have the name of their home market in their fleet name as if it were their faction name.
Scavengers fall in a grey area between independents and pirates. They only attack in defense of themselves and their property, but they have a looser definition of what is theirs than other people. If they see an unknown (or known and not hostile) fleet near "their" salvage then they'll turn on their transponder, intercept it, and demand it leave the area. If it's another scavenger the weaker fleet will scram, but if they are similar in strength (which scavengers are not the best at judging) they will fight it out. If they catch a fleet nosing about more than once they skip straight to shooting (if "even" or stronger) or (if weaker) demand the offending fleet leave even more aggresively (while planning their escape route). In any case, getting in a fight with them has little affect on the player's reputation.
I feel like small scavenger fleets should be a lot more common than the giant ones we usually see. Maybe scavengers should be pirates and only spawn in the fringes, while the big scavenger fleets that fly out and explore are independent "tech miners".
Mercenary Patrols and the like mostly act like regular patrols, except they just attack the player when they catch them transponder-off. However, they will accept a communication request if the player initiates it - at that point they act like a regular patrol. Mercenary patrols can be independent or part of a faction. Faction mercenary patrols mostly spawn at low-stability markets because they're cheaper than a proper navy.
