I dunno about lowering ship performance beyond the fact that it becomes easier to damage engines and weapons once armor is down, but I'd appreciate it if damaged ships got even cooler graphics down the line.
Rather than just the shattered look it has now actual parts would break off, possibly (but not necessarily) modifying the hit box. Occasional flares or additional explosions and crew members could be seen blowing out into space, use off the engine would result in flames coming out the sides of your craft, etc...
Just additional, random graphical doodads to make the damage look more dynamic. Like... holy crap, this ship is seriously falling apart on a structural level, I should probably do something about that.