Pather attacks are rare. You lose more money by not using the item than by accepting occasional attacks. Its not worth worrying or considering pather interest when building colonies IMO. Its just sometimes a money booster to go pop their base.
For defense against core: I've found star fortresses + patrols to be sufficient so far, especially as its now possible to boost defenses very high with a bit of item luck and doing the red planet quest. Even if the enemies destroy the orbital station, it will regrow, and their bombardments never succeed. A military base in system does make things safer and stability higher by protecting the comm relay, but I've found that to be a luxury rather than a necessity. So... I don't really think improved military or bombing core worlds are needed to keep colonies safe, once a colony is up and large enough to have full defenses.
Stability is a fair concern, especially as how drastically commerce boosts income (especially with an item). Keeping stability high is a priority, and sometime comm relay + ground defenses + orbital fortress isn't enough. I haven't needed to use any story points to keep my colonies at 10 stability except when under the 3rd level pirate raid, but I could see if I had decivilized and maybe a popped starbase and resource convoys destroyed from an expedition that it would dip down to 5, which is not ideal.
As an example, I have a good colony in my current save very close to the core: size 5, 125% hazard, +0 organics +2 ore +1 rare ore, mining, commerce with an item and improved, spaceport with an item, refining, defenses at several thousand, and a 1 skill industry admin. It makes about 200k/month, with 2 story points invested, and sits at 10 stability with 2 in reserve. This isn't an optimal colony: only 2 items, using an admin instead of an alpha core, no cores on industries, no items on 2 of its industries. Its accessibility is high thanks to being very close, which is nice. But its still a single world making me 200k per month, which is plenty profitable without needing to put more points in. More points would be more money, but I don't need more money, so I can spend my points on other things.