I would like to say one thing right off the bat: Teleporters are too startreky, and any mention of them will *** me off.
Furthermore, for a space age where the enemy has the ability to teleport anywhere into your ship. The federation had a pathetic interal defense system, and is not a good example. (I still remember that episode where someone hacked into the computer system in the galley and ensign Kim yells on the Bridge: "I'm locked out!") Seriously people.
Qloos' Proposition
All ships have a crew capacity, if the crew is killed: A ship is disabled. This crew can be swapped out for military personel. Too many military personel can reduce the performance of a ship.
Boarding option number 1: Ram him, attach the ships and cut a hole through the hull, (and armor if applicable - which would take longer.) This is best done when the ship is overloaded from flux and is best NOT done when a ship is venting flux - which gives venting flux a defensive attribute. At the very least, it is opportune to attempt the docking when the enemy does not have an engine boost. Because if the enemy is moving fast, there is less of a chance of a successful dock.
Balance between large and small ships would be quite simple in this case. Faster ships evade capture, larger ships use a large security force to battle back.
The downside is you have to close in to dock with the enemy and risk being fired upon, also the shields on both ships would have to be down, (atleast on the side where the docking is taking place, otherwise the ships would repulse each other.) The pros would be a direct connection for your forces to move onto the enemy vessel, allowing you to bring your full force to bear at once.
Boarding option number 2: Shuttles, (similar to the commanders.) Would attempt to fly over and dock. They would attempt to fly around omni directional shields and wouldn't be hindered by gunfire once they're docked safely on the outer armor. That is - once they're docked. As they fly between ships they would be vulnerable to the numberous point defenses that would target the shuttles like any inbound missile. Also, shuttles cost money. So the pros of this option would be keeping your main ship out of harms way, while the risk is entirely to the boarding parties.
Boarding option number 3: EVA entrance. Why waste money with shuttles? Stick your boarding parties in space suits with grappling hooks and shove them out an airlock. Maybe they'll get lucky. One hit their dead, but when you're desperate you're desperate.
Boarding option number 4: Ship to ship tether, with a secure connection, your milita can slide along to the othership. Maybe I'm getting carried away with my suggestions now, I'll stop.
Once an enemy force is on board, the action can be seen in an already featured display window! With the ship armor status screen, squares could equate to ship sections, and could be shown in red and green for which side controls what. If a ship section is hit during combat, security foces/boarding parties in that area could be killed. Why stop there? Click on the ship overlay to order your security foces to concentrate at a specific section of the ship in preperation for boarding action or defense!