Recently I've been playing with the limitation that my fleet must stay small, so I've forced myself to play better.
There are two main issues with playing the way I do.
Firstly, enemies regularly outnumber me. This is fine if I just optimize my ships more, but against tanky enemies it can get difficult.
Secondly, AI is extremely stupid when it comes to fleet cohesion. Any ship significantly faster than the rest will immediately get isolated and die. Even a hammerhead is fast enough for this effect.
You simply need tougher ships. Look to add some really tanky ships into your gameplay. If you are not going to win by numbers you have to win by brute force. These are typically high tech ships like the Paragon and if you really don't want a capital then a pair of Apogees with hardened shields and officers with the right skills are what you need.
Avoid ships with burn drive, they are more in tune with overwhelming the enemy and killing them faster than they can organize. This play-style is effective against equal or lesser numbers. As soon as you get a Dominator it will just burn-drive to its death especially if most of your other ships are surrounded in a ball formation.
If you want to use missiles and fighters then you have to use them both and ideally in replenish-able numbers. This means pilums with bomber squadrons to put so much point offense on field they can't all be shot down, or using a bunch of effective killing missiles like harpoons on griffons for their missile auto forge ability.
If not then you need to have ships that can get in and out of combat reliably without getting killed. Your options are limited to Eagles Falcons and Auroras. They all work well together or in numbers of individual types.
Destroyer and frigate size variants are ill advised unless you use them for a very specific backline support like a buffalo mk II with pilum launchers for their low deployment cost.
Tell me what you would prefer and I'll help guide you into a small fleet that can effectively take on much larger foes.
Everyone here talking about how to take advantage of the game mechanics and no one is actually thinking of how the ships themselves work.