Diktat, League, and Tri-Tachyon always want to wreck your colonies unless they produce (almost) nothing. Once my colonies grow big enough (about size 5 or 6), it is too hard to avoid expeditions. Open Free Port, and Church and big H join in with the bullying.
Sure Diktat, League, and TT do not care about Free Port, but it does not matter. They attack you anyway if your colony is bigger than a fledgling and has an industry.
Once I open Free Port to significantly boost population growth of my colonies and maybe feed miners' drug habits, expeditions become annoyingly frequent, and the urge to kill all of the core worlds (or let the pirates do it for me) to make them stop rises.
Basically, Free Port status is choosing between three factions attacking you somewhat infrequently (off) vs. five factions attacking you often (on).
"Once your colonies have Free Port, all of the major factions want to wreck your stuff anyway..." is true enough eventually, even if not precise enough, because those majors who do not care about free port are attacking you anyway. (It is very easy to have too much market share at some point.)