When you don't have a fleet big enough to start proper engagements, leave some of your ships uncrewed. The ones you don't want to get caught in fights, move them as close to the bottom right of the fleet window as you can and then run short on crew. Uncrewed ships won't go into battle, obviously. However, if you have to pull a hasty retreat for any reason, don't expect to be keeping those uncrewed ships either.