It's biggest weakness is that the poor Brawler constantly has it's butt hanging out in the breeze for missiles and fighters to hit. I usually take so much engine damage that, in addition to the slow speed, I can never catch anything.
However, if you use it as part of a fleet, like you said, it can excel.
It's two medium Ballistics give it a LOT of versatility that most folks overlook. Everyone just slaps the dual Assault Chainguns there and figures that's it.
It can fit some awesome shield-destroying weaponry there easily. Heavy MGs, Heavy Needlers, and probably my favorite, Hypervelocity Drivers. A Brawler with dual Hypervelocity Drivers is a beast. And the other two slots are universal, so you can fit beams, missiles, or additional ballistics.
The problem is survivability. Like I said, any attacks from behind screw you over and almost every ship in the game can disengage from you at will due to the Brawler's low speed.