I did a rather lot of testing on this subject recently, and my findings are thus: The only thing that really matters is the disposition of the enemy fleet. If you attack, and they escape, any ship not in the fight when you win surrenders. I got 3 onslaughts this way.
As for actual boarding operations, I spent a few hours trying to kill stuff very softly. Nothing seems to matter. I tried not using missiles, I tried not using kinetics, I tried not using beams. I tried focusing 100% of all fire into one spot on the ship. I tried spreading the impacts all over the armor. As best I can tell, if any of that has any effect, it's not a very big effect.
One thing has popped up as interesting though, It appears that fighters and bombers that survive being disabled, but have only 1/3 their total ships left, are almost always boardable. IE: Thunder bombers get disabled, but 1 of them isn't destroyed. After the battle you have a superb chance of boarding this ship. Which kinda makes sense. If you were the pilot of a fighter, disabled and your carrier fleet is dead... you'd ask for a ride too.