It's not just anti-fighter, it's anti-armour damage that hits more often than Harpoons, and a nearly continuous stream of PD-overwhelming, and AI-confusing missiles. Annihilators are great if you have a slower ship with good missile turrets. Ballistic alternatives are great if you actually have the slots/flux for them. On many frigates and destroyers, you simply don't.
Annihilators happen to be better anti-armor damage - you only need to hit 40% accuracy to deal more damage than perfect accuracy Swarmers, which isn't
that big an ask (though obviously they don't have tracking) and they're better at overwhelming PD (you don't need to even hit to affect PD, just get in the weapon arc). For the record I don't even like Annihilators that much, but Swarmer's manage to be worse in the things that matter.
A Swarmer has less HE DPS than a Light Mortar while also happening to take a more important weapon slot and costing more OP. No ballistics-using ship larger than a Kite is struggling enough with HE damage that Swarmers will make the difference, and ships without ballistics almost certainly would rather have a kinetic missile (if you're fighting slow shieldless ships, any other HE missile would be better and if you're fighting fast ones then Salamanders or Atroposes would outperform them).
Economically, they are 4 OP and can fire in any direction rather than a fixed arc. PD Lasers and Vulcans are 4 OP and fire in a fixed arc. Using my brawling simulator (which assumes 100% accuracy), these are the times it takes to kill a Broadsword:
PD Laser: 14.46 s
LMG: 12.81 s
Vulcan: 7.26 s
Swarmer: 11.33 s
Perfect accuracy is wildly optimistic for LMGs and Vulcans, of course; in practice they might hit 30% of the time and the target will only be in their arc occasionally. And sometimes there's a pretty big opportunity cost to dedicating kinetic/energy slots to PD while missile slots can be less so (especially if they are dual purpose!).
Swarmers also have to have the correct target: if the ship carrying them is targeting a ship, it will ignore the fighters; if it isn't targeting anything at all they'll just go for the nearest target--which may also just end up being Decoy Flares! And they have 50 projectile HP, so they can be shot down by random fire. And they have limited ammo.
As an actual in-simulator example, an Enforcer with 4 Swarmers and 4 Vulcans is barely more effective than a 4 Reapers, 4 Vulcans and IPDAI against a 2 Broadsword, Twin Salamander EDC Condor (87 vs 92% replacement) by the time the Swarmers run out of ammo. But the latter case gives you four Reapers and a very useful hullmod.
The opportunity cost of PD weapons is nearly always compared to leaving a weapon slot empty, since most ships don't have the flux to kit their entire ship in proper fighting weapons anyways. Swarmers are competing with Reapers/Hammers/Atroposes, Sabots and Harpoons. The opportunity cost not in the Swarmer's favor.
Swarmers aren't great anti-fighter. I'd say their 'primary' role at this point is anti-frigate armor and they are good at that - not really any need to mount HE guns for that kind of fight. But they do help against fighters, and frigates need all the help they can get.
Let be honest, two fighter wings just plain wins against a lone frigate unless its anti-fighter or high performance (Omen, Tempest, phase). A wolf was used as an example earlier: 2 Talon wings will rip a wolf down in seconds. With swarmers the wolf does a little bit more damage before horribly dying.
If its the ship next to the wolf getting attacked however, the swarmers roughly double the anti-fighter DPS the wolf is giving as support, assuming its equipped with a pair of LRPD lasers for PD. Its still not great, but its better than without the missiles at all.
If a Wolf were in a situation where it was getting eaten alive by Talons, doing a little more damage is a moot point because there's no way the Swarmers are inflicting enough damage to have a notable impact on replacement rate.
Rather than asking, "how can I change my Wolf from being abysmal anti-fighter to merely bad/mediocre?" maybe it's better to ask "how can I change my Wolf to be better at distraction and opportunity attacks (which Wolves can actually be good at)?" When it comes to missiles, any other option is a better answer than Swarmers.
They aren't so much better at their anti-frigate armor niche (something that is also covered by a dozen other weapons) that they're worth using over other missiles which can fulfill more important niches.
