Unlearning the skill does not remove the third admin from a colony. He can be moved around by moving another admin in his place, and he will gladly swap planets. (I was wrong about him not being moved before.) Not being able to add him to a planet to administer means nothing when he is already entrenched on a planet before unlearning the skill.
Even with cores, I might consider this exploit as one less world to deal with foreign interference. That is, get three admins with Colony Management then forget the skill.
Either Colony Management should be a permanent skill, or it should forcibly remove the third admin from the planet he is ruling over.