To a large extent, yeah. I'm bumping into this as I'm trying to fight endgame enemies with a primarily low-tech fleet built around Shield Shunt and Polarized Armor. I'm still able to win in most cases, but I feel like the difficulty differential between normal enemies (pirates, intel bounties, etc.) and top-tier Remnant/Dorito fights is significantly higher than it is when I play with a more standard, shield-centric fleet. My armor-tank fleet is decently lethal (love being able to dedicate all flux to weapons) and handles the smaller fry just fine, including mid-tier Remnants. However, when going up against Radiants packing 3+ Tach Lances in fleets with full Alpha Core officers, things get dicey.
Relying on armor makes fights into a race - you need to overwhelm your opponent before you get ground down - but it's far more difficult to come out ahead when you're facing those anti-armor large energy weapons mounted on ships that rival your own for durability. Low tech's lack of mobility hurts here too, since the very toughest enemies are also extremely adept at darting away to vent in safety and my ability to pursue them is limited. If I can't get some hits in while they vent, they come back completely fresh while my ships are still carved up from the first exchange. Then I'm well behind in the race.
So ultimately it seems to me that dedicated armor tanking is a bit of a win-more mechanic. It makes it easier to defeat weaker enemies, and harder to take on the most powerful. Kind of a bummer, because I'm otherwise enjoying building and running this fleet. It's a nice change of pace in combat and I like how the AI behaves with shieldless ships. It also turns ship construction into a very different kind of puzzle, where you're balancing a different set of considerations, using familiar tools in new ways. Hullmods that used to feel marginal are now high-priority, point defense becomes a much bigger consideration, and so on. Plus it turns out that it's really fun to be able to shoot all your guns all the time.
On the Vanguard in particular: Naturally I tried using it in my fleet since it fits the theme, but I actually gave up on it pretty early on, even before running into the endgame threats. I guess it's not surprising that the smaller the ship is, the less viable it is as an armor tank. I also think it suffers a lot from having a Damper Field and no shields (AI is braindead about this), and the expense of recovering it takes away a lot of the appeal of Rugged Construction - both of which people have discussed already in this thread.
On top of that, I also find the Vanguard hard to outfit in a satisfactory way. It has way more mounts than flux capacity, so you want to put missiles in the composite mounts. Those are also the only hardpoints though, so if you want to use any high-recoil ballistics you're stuck putting them in turrets instead. And it's OP-hungry for all the usual armor-tank hullmods, but also badly wants the missile mods. In the end I just couldn't find a build for it that felt right.
It's a fun idea, and I'd like to have a frigate to round out my armor-themed fleet, but I don't think the Vanguard is there yet. It needs some more work to feel like a good pick, especially out of the early/mid game.