So, your problem is that you don't have enough RAM *left over* for the graphics assets. The way VRAM works is, for most drivers, the texture assets are resident in both system RAM and video RAM. If you're forcefully allocating 10 GB of system RAM for starsector, that leaves just 6 GB for the operating system and all other components, including the graphics driver.
As a result, you probably have less than your stated 4 GB of usable VRAM... in a manner of speaking. The way it really works is quite a bit more complicated than that, but not really in the scope of a post here.