Fighter swarms would probably be less effective if, say, there were more PD weapons with AOE effects that could hurt fighters particularly well, the same way a Doom can utterly gut fighter spam: All the fighters stack, the mine goes off, they all die.
Of course, ultimately, I know of no game where fighters are ever "balanced". In every game, ever, fighters come in two flavors: Useless, or, if they can be massed at levels which hit a critical mass, overpowering. All swarm units in every game, and most likely in real life, function like this: They're either useless and can be swatted down before they inflict real damage, or they hit critical mass and can no longer be brought down before they cause real pain, at which point it turns into a case of "AAAH, BEES!". And there is ultimately, no functional distinction between fighters and missiles. In any game where one is OP, the other will almost certainly be also. Ultimately, a fighter is just a missile with 4x the delta-V it really needs.