Ok, so imagine earth was a size 6 colony with decivilized subpopulation.
You have ~an NYC sized population center, with industries and military areas spread around the planet.
That's still going to be a lot of space for decivilized people.
Granted, once you get beyond size 6, it becomes less and less reasonable.