Multiple colonies within the same system with Fleet Commands all add to the defense, IIRC, so you're not limited to just the defense that Colony X can put out.
They shouldn't even need full Fleet Commands - Patrol Bases will help too. Similarly if you're using Industrial Evolution, then Military Relays will do the trick too.
Now, for minmaxing Fleet Strength-
First, you'll need a Very Hot world (so lava type, probably, or I believe you can use an orbital lamp to make a hot world very hot?). This will be where you put your Fleet Command, with a Cryoarithmetic Engine.
Second, you'll need basically any world for your Orbital Works, where you'll want a Nanoforge - at least Corrupted, preferably Pristine (you can raid one from the League or the Hegemony). This can be the same world as your Fleet HQ, but you'll probably be burning some cash on hazard pay to get that to grow.
Slap in the usual array of ground defenses, planetary shields, and cores, then - most importantly! - remember to set your defenders to resist. Don't be the guy who leaves them on comply and wonders why the orange gits stole all your cores.
Alpha Cores in the Orbital Works and Fleet HQ (and, well, everything else) will beef up their odds of survival. If you're like me and running Terraforming & Station Construction, throwing down a trio of orbitals with military relays will also ensure that there's a massive swarm of defenders around your 'homeworld' at basically all times, and given the inspection fleets can get strung out by delays and hyperstorms, their odds of survival can be pretty slim;
Spoiler
(Though that's actually a Persean trade fleet, I think. Still, it gets the point across.)
Even without mods I'm sure you can get to the point where your colonies don't need constant babysitting to fend off AI inspections. It just might be a bit more challenging! But eventually you should hit a critical tipping point at which point they can't do anything serious to you. You'll have to pick a good system though.
Also, hang onto a couple of story points at all times - that way you can always use them as a 'get out of inspection' card by bribing the officials to look the other way. It's pocket change compared to what you can earn from missions/bounties, and it'll buy you a couple more years to get fully established before you reveal your fully operational battle station.