Speaking of that, another related issue is if a faction successfully invades and takes over one of your colonies, you can just go to that colony and re-invade, and you nulled the entire purpose of the enemy invasion.
The fix I propose, that isn't too intensive (like having defense fleets hover for a long time), is that the "invade" option in military options is faded out, and when you hover over it, it reads "it is impossible to raid this colony; the current leaders of the colony have heavily fortified it due to unrest in the local region, and local patrols are upgraded to deal with a possible retaliation by the original owners. The traffic and defenses will die down in 365 days").
This fixes invasions not being a hassle at all, and makes spending money on a invasion fleet or invading the port of origin of the invasion fleet yourself a more preferable option than to just let your colony get taken over, since you will lose 12 months worth of income from it (and if it gives you 200k or more a month, thats a lot of income lost).