The Seeker Mod by Tart kind of does some things like that, and I know that there is a bio-alien faction in the works, as well as biomechanoids.
However, i had an idea for three factions that I probably won't be able to get around to.
1) The Jellyfish faction
It's essentially jellyfish alien monsters that spawn from random planets in uninhabited systems and don't do much other than float around and not do much, with some aggressive ones that target random shipping. Idea is like there's a 'passive' faction that won't attack until you do, and one that will ALWAYS attack you. Killing them drops Monster Bits, which is very similar to AI cores. They use burst beam weaponry, as well as using tentacles to attack (a la the Angel of Might from Evangelion Rebuild) along with swarming fighters. No shields, but regen armour and hull.
2) The Geometry Faction
Alien monster faction (again), except it's geometry. So there's a circle one that shoots antimatter shots, a triangle one that shoots beams from it's corners, a square one that is hard to kill, a pentagon one, etc. However, they have alternate forms - the circle one folds into itself and phases, the triangle one turns into a fracal - and shoots from it's corners, the square one folds open and unleashes a hellish beam attack but also becomes vulnerable (Ramiel from Evangelion Rebuild), and the pentagon one sort of flips into itself and does an AOE attack that wipes out fighters, missiles and deals a sledge of damage to anything too close to it. They float around hyperspace, and are more common near the edges of the map. Drop hyperspace Bits that also work like AI cores.
3) the Cthulhu Faction
The idea is this: A boss faction where the ships have no weapons, instead using drones with shields and guns on them. The ships phase, but the weapon drones don't. The ships buck out of phase when all the drones are killed, and remains out of phase until all the drone weapons are rebuilt, before phasing again. Also has infinite CR. Ships look suitably biomechanically alien, like sealife crossed with Giger. Definitely boss faction, but shows up when the player reaches near the edge of the map.