Use marines to board ships that rarely appear in shops, or anything that is good if you only see ten men or less. In other words, stuff that you would give everything to get a ship you covet or when risk-to-reward seems to be in your favor.
Always, ALWAYS hard dock. Chances of successful capture with hard docking is unreliably low enough already. No need to make it even worse by launching assault teams.
Keep boarding in mind when building your fleet. If your fleet has big ships, use them to board smaller ships; they can survive an explosion. If you want a fast fleet or you think you might board a Paragon or other big ship with huge flux pool, bring some Mercury and/or Hound ships. Mercury is much more common in 0.65.
For marines, bring at least ten. That will be enough to board many ships. If you want to board anything, bring up to 200 (or less if you have Special Ops perk.)
Remember, explosion damage varies by the ship's flux capacity. See if your ships have more hull and armor. If so, use it to board the ship. If not, use your disposable Mercury ships.