In my current game, I was almost ready to colonize my desert system next to core worlds until I noticed three or so stable debris fields slowed down the game.
I'm sorry, I'm confused by what you mean with "slowing down the game". Are you talking about performance issues? I didn't think about that at all considering I never had any. I thought you were put off by the small travel speed debuff of asteroid belts and stuff. Gameplay wise I never lost against any expedition even with a single planet in a system, my patrols always win, I don't even need to be there (vanilla game, no mods); I'm only bothered with the visuals.
I'm not talking about making every system home to 6 planets with 3 gas giants, rings, moons and stuff. I'm talking about making those systems a little more common; from my experience, a good half of the sector is exactly what you want, almost empty systems with maybe five astronomical objects in it.