My (offline) machine does not have Java-64, so I am stuck with whatever the Starsector installer provides. I have not tried to allocate more than 1 GB recently, though when I tried before, it would not work. My machine has 8 GB and 64-bit OS, so I would allocate more if I could.
I do not have a gpu, so ShaderLib effects are disabled (but the mod is enabled for mods that need it, like Blackrock).
When I run Nexerelin, the only faction I tried so far that did not have problems was Knights Templar. Blackrock loads, but sooner or later, my game hangs or crashes. I tried loading Valkyrians but cannot, even with no other unnecessary mods. Loading bar goes all the way, but screen never changes.
So far, if I want to use mods, I load very few at a time.