In my opinion, going with physics on this one makes for better and funner gameplay balance.
What do you mean by going with physics? There's arguments to be had for fast small ships and slow big ships, for rapidly-accelerating but low maximum speed small ships and slowly-accelerating but high maximum speed large ships, for quickly accelerating small ships and slowly accelerating big ships with the same maximum speed, and so on.
If large and small ships can turn fuel into kinetic energy with the same efficiency, then if a given mass fraction worth of fuel can accelerate a little ship from V1 to V2, that same mass fraction of fuel should be able to accelerate the large ship from V1 to V2, and the maximum practically attainable speed of the ships will be determined by how rapidly the ships can accelerate (determined by mass:thrust ratio and any limits imposed by structural strength, equipment acceleration tolerance, and crew/passenger acceleration tolerance), and for how long the ships can accelerate (determined by fuel supply and thermal constraints).
If there is a size dependency on the efficiency with which ships can turn fuel into kinetic energy, that will tend to push the higher maximum practical speed towards whichever end has the better conversion efficiency.
If we're not dealing with magical technology able to dispose of any amount of waste heat, then larger vessels should tend to have more thermal issues than smaller vessels; total heat generation should be roughly proportional to internal volume, whereas total heat dissipation should be roughly proportional to surface area. Big ships may not be able to have reactors as large relative to the ship as the small ships are, or may have to use relatively less powerful thrusters, or some such thing to keep the heating issues manageable. Or they'll need to have cooling systems which are in some way better than those of smaller vessels, which means that the cooling systems may well be relatively larger.
Due to structural stresses, large vessels may well need to dedicate a larger fraction of their volume to their structure than smaller vessels would even if the larger vessels have lower accelerations, reducing the space available for all other purposes, including fuel and propulsion. Structural limitations can also come into play against smaller vessels; there's a limit to just how small you can make the structural members if you're going to be handling a given amount of force.
There's also the potential for space issues, which might for example result in a small vessel trading a bit of fuel efficiency to keep the thrust of the drive engines at adequate levels for the desired performance.
Then there's the question of whether or not it's economical. Maybe you can make a big ship with the same fraction of its internal volume dedicated to structure as a smaller vessel, but you'd have to use a more expensive material in order to do so. Maybe you can make little reactors and thrusters as efficient at converting fuel to energy as the big reactors and thrusters are, but it costs an arm and a leg. Maybe you can make a cooling system that'll handle the waste heat of the big ship well enough to give it the power it needs to have the same performance as the little ship, but you can give up a little of the ship's performance and gain a great savings in cost.