Bounty payout being worth it depends heavily on how concentrated and powerful the player fleet is and to be honest on player skill, both in combat and in fleet building. 300k multi capital bounties can be done with a couple of cruisers + half a dozen destroyers + support ships (and not current power strategies of Dooms/phase ships/Derelict contingent either, just normal non-SO ships) because they aren't nearly as dangerous as their ship numbers suggest. They tend to have low quality loadouts and officers, at least compared to endgame bounties.
My first piece of advise would be to audit the fleet and figure out what ships are needed and what are wasted. Having a decent number of ships on the field is really important to stop from getting swarmed and distracted; having deadly ships on the field to quickly kill enemy capitals by means of force concentration is really important. Those 2 aspects need to be balanced in a maximum budget of 240 DP, with how battles and skills work at present. Personally, I have found for current balance that either 1 or 2 capital ships (or equivalents) is the right number and more is sub optimal. Too many cruisers is also bad: gotta have a decent number of ships! The fleet you mentioned - 5 battleships, 6 cruisers, 8 destroyers, etc - is so large that even half of it can't all be deployed at once, its losing fleetwide bonuses so every ship is weaker than it could be, and many ships probably do not have officers unless the player is spending cash and story points (and real world grinding time) on mercenaries. It also really expensive to maintain: a lean, concentrated fleet can scavenge twice the supplies it spends fighting from the enemies it kills without issue (though having a few shepherds along to boost post battle salvage helps there).
My second piece of advice: have second line ships and treat them as such. Its true that bounty payouts for endgame fleets are less than the cost of repairing a capital, and even cruisers are pretty expensive. But destroyers and frigates are cheap and its also not the end of the world if they get a D mod or 2 from being blown up: they aren't meant to be high performance killers like officered ships, they are there to act as support ships to the officered ones. When its acceptable for a cheap ship to be blown up 2 or 3 times before they have enough D mods to be truly worthless, the marginal loss starts to get really low. Have one of these second line ships set to escort each large ship that you don't want to get blown up, and make sure that the fleet setting for non-officered ships is aggressive or higher (3 ticks). Escorts with these setting will swing around to the "front flank" area of their escortee ship and take a lot of pressure off, even if they aren't laying down all that much fire. Most of the time losses will come from these low performance, cheap, and disposable support ships rather than the expensive ones.
I wonder if it would be better if random bounties didn't go over a threshold (~150-200k maybe) and high end bounties were exclusive to contact missions.
I would not like this. 150-200k are fights on the level of first few destroyer purchases or even tutorial fleets (for the weak pirate ones, bit of RNG there) for player who specialize in bounty hunting. I like the contact system in general, but I don't want to have to grind through 10-20 bar missions to get a decently important bounty contact.