I think the idea of "free port" needs to be expanded into this suggestion. No longer would it be just a "trade all goods" button combined with growth incentives/stability modifier, but it would become a "select legal trade goods" button. This would impact profits, growth rates and reputation with other factions, based on their access rating with that colony.
Every time a faction received legal (to you and to them) goods from your colony there would be a chance for your reputation to slightly improve. Whenever they received a shipment of goods legal to you but illegal to them there would be a slight chance for reputation to decrease.
And that is just the most basic implementation. I would love to see a more detailed system that let you set "locally produced/used legal", "import legal", "export legal" and "all legal", with various impacts associated with each. And these effects should be unique to each good (e.g. allowing export of harvested organs should increase profits, but actually decrease population growth and stability, whereas only allowing their import or local use should decrease profits but increase population growth. Allowing local use but not exports should lessen the rep hit for black market exports to another faction, but not eliminate them, whereas making a good 100% illegal in a colony should eliminate reputation hits with another faction even if illegal trade is still technically happening)
"accessibility" would no longer be tied directly to the "port status", but instead would be dynamically attached to how "desirable" your port looks to traders based on your market settings. Not having a good set as legal to trade in some way would not have a huge impact if traders were not looking for that good from your colony, but could have a much larger impact if they were (e.g. why would they care if raw ore is or is not legal at your port if your port doe not produce or sell that commodity? on the other hand your colony does produce a lot of drugs, so making those illegal will mean they are likely to look elsewhere for a less risky supply)
And tying into that mechanic, the player should be able to set the tariffs on individual commodities too, both coming and going. This would impact access as well, as it too would dictate the desirability of merchants to trade at your port.