Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Schwartz]

Maybe. Throughput on a fast-ish HDD should be at least 80MB/s in either direction, so that would make up all of about 5 seconds to pull the data off the drive. I assume it has to do with it being 10k files (mod directory) as opposed to the 2.8k of Starsector itself, and SSDs being so much better at random access. Would be nice if there was a way to just shove all the data into memory first and then put it together from there. Or not front-load the whole game, but I don't know how much of a pain to implement that would be.

[Alex]

Right, yeah, it'd be about the access pattern and not straight throughput. I wonder if defragging might help.

For reference, on my dev box (with a 4-year-old sandisk SSD) vanilla takes 8 seconds to load without sound and 13 seconds with. Will look into this, at least a bit.

Would be nice if there was a way to just shove all the data into memory first and then put it together from there.

Not sure that'd help, if the issue is collating the data from disk in the first place. But it could be something unexpected.

Or not front-load the whole game, but I don't know how much of a pain to implement that would be.

Pretty huge.

... and then another 1:30 of 'Cleaning buffer' messages in the log.

Fairly sure this is ShaderLib loading its normal maps, since the only hook it can do this at is after the loading bar looks completed. Depending on whether the extra load time is worth the improved graphics, turning that off might help provided it actually doesn't load those if disabled.

[TJJ]

It's seek time/latency that dominate IOPS performance when dealing with small files; hence why SSDs perform so much better (A typical SSD will be ~50x faster than an HDD for reading 4KB files.)

Threading the loading can see even greater gains for SSDs, as they're highly parallel (particularly performance-grade models), able to service many I/O requests simultaneously.
The common comparison is 64 simultaneous 4KB reads (NCQ=64); where speeds ~200x that of HDDs are possible.

That's why most games adopt a packed file format of one kind or another, with the data logically ordered to aid reading; it doesn't help SSDs much, but makes a world of difference for HDDs.

Packing the resources in Starsector might be worth looking at in the future; though to see significant gains, you'd need some kind of usage analysis to determine how best to order the pack file. (blindly sticking it all in a zip/jar would help.... but not much if your load ordering causes skipping all over the archive)

Though packing would probably complicate modding a little.....

OP's computer seems pretty dated too, which won't help the situation =/

[Midnight Kitsune]

That's so strange. I've run the game with maybe 10 or so mods while loading up modded saves to reproduce a bug, and it's never taken *that* long - maybe a minute at the most. I wonder if an SSD makes this much of a difference. Will have to look at it at some point, I suppose.

Yes it does, as do faster HDD speeds. I know as I switched from a faster HDD to a slower one when I got my new craptop and I noticed an increase in loading times
Also another thing: I notice that my bar takes quite a bit (about 30 to 45 seconds?) to START filling and during that time SS will lock up and behave as an unresponsive program. But once the bar starts to move, it is fine

[Dark.Revenant]

... and then another 1:30 of 'Cleaning buffer' messages in the log.

Fairly sure this is ShaderLib loading its normal maps, since the only hook it can do this at is after the loading bar looks completed. Depending on whether the extra load time is worth the improved graphics, turning that off might help provided it actually doesn't load those if disabled.

GraphicsLib increases loading times by about 15-30%.

[Schwartz]

Can confirm what Kitsune said.

Whichever way you can deal with the many tiny files problem, that'd help loading times greatly I think. I'm always keeping my disks defragged and the game runs off a pretty fast Seagate 7200.12 as well, but that still doesn't help matters much.

[MesoTroniK]

One thing I noticed on my end, is that after Alex redid the launcher that game load speeds got two to three times faster.

I do have a SSD and a fairly high end rig in general.*

[Elaron]

I just did a few tests on my secondary box to see how much effect a hard drive would have on launch times for a heavily modded StarSector installation. The full details are inside the spoiler tag, but the TL:DR seems to be that, as long as you have a reasonably modern hard drive, launch times are not hard drive bound.

I watched the disk queue length, active time, and data transferred metrics in Resource Monitor while each test ran, and while the old HDD definitely struggled, neither the modern HDD or the SSD seemed particularly stretched.

Spoiler

Hardware:
CPU: Core i7 965 @ 3.2GHz
Memory: 12GB
Motherboard: Asus P6T Deluxe (SATA 3Gb maximum)
GPU: GeForce GTX 570
Drive 1 (SSD): Samsung 850 EVO 256GB SSD
Drive 2 (Newer HDD): WD Blue 1TB (64MB cache, 7200 spin speed, NCQ)
Drive 3 (Older HDD): Seagate 7200.7 160GB (8MB cache, 7200 spin speed, NCQ, SATA 1.5Gb)

No defragmentation was run on the hard drives after Starsector was copied to them, but they were both > 80% free. Fragmentation is unlikely to be a concern.

Operating system: Windows 10 Home 64 bit

StarSector Setup:
Version: 0.72
JVM: 1.8.0_111 64 bit (Yes, I know Java 7 is the recommended version, but this has been stable for me so far)
XMS: 8192m
XMX: 8192m
Script caching: Tried with True (cache already compiled), and False (cache deleted), but made no difference.
Mod folder has 14,582 files, 1,932 folders, and 513MB in size
Mod list:

Spoiler

StarSector+
Dynasector
AI Battles
Autonomous Ships
Autosave
Blackrock Driveyards
Combat Chatter
Common Radar
Console Commands
Dassault-Mikoyan Engineering
Diable Avionics
ElizaLib
Interstellar Imperium
Junk Pirates / ASP / PACK
LazyLib
Metelson Industries
Nexelerin
Portrait Pack
Practice Targets
Save Transfer
Scy Nation
Shadowyards
Ship/Weapon Pack
Simulator Overhaul
Steiner Foundation
The Knights Templar
The Mayorate
Tiandong Heavy Industries
Torchships
TwigLib
Underworld
Upgraded Rotary Weapons
Version Checker
Audio Plus
GraphicsLib
Lightshow for SS+

[close]

Test setup:

• Restart Windows. (This step is important to clear Windows' in-memory cache of recently used program files)
• Wait until CPU / HDD has settled down.
• Launch StarSector from drive selected for the test.
• Timing is taken from pressing "Play Starsector" in the launcher to the appearance of the main menu.
• One test per set per drive (all I had time for really)

Results:

• Drive 1 (SSD): 1m 30s
• Drive 2 (Newer HDD): 1m 35s
• Drive 3 (Older HDD): 2m 10s

[close]

[Schwartz]

I have to correct my earlier benchmark and I figured out why it took so long on my setup, too.

When you start SS, it writes and deletes files in rapid succession in Appdata\Local\Temp, which I happened to have encrypted with EFS. When I disabled that, loading times improved from 4:00 to about 1:40.

This could also mean that if you have SS on the same drive (not partition) as the Temp folder, and the two of them are reasonably far apart physically, the drive will have to seek back and forth a ton which costs you time.

Workaround would be to logically link the Temp folder up to work off a different drive - or move SS to one. But I'm not sure at how much of a gain you're looking at here. It should also be said that any kind of disk encryption will incur a performance penalty. EFS is just particularly bad at handling a lot of small files.

[Elaron]

When you start SS, it writes and deletes files in rapid succession in Appdata\Local\Temp,

That explains the disk activity I saw on my system disk (the SSD) during my tests. I expect people with a standard single drive setup will see more of a difference between SSD and HD due to the extra write work.

This could also mean that if you have SS on the same drive (not partition) as the Temp folder, and the two of them are reasonably far apart physically, the drive will have to seek back and forth a ton which costs you time.

Even being on the same partition will cause a performance impact, it'll just be the nature of the beast.

Workaround would be to logically link the Temp folder up to work off a different drive - or move SS to one. But I'm not sure at how much of a gain you're looking at here. It should also be said that any kind of disk encryption will incur a performance penalty. EFS is just particularly bad at handling a lot of small files.

I'm going to have to think of a good way to test that with the computers I have to hand.

[TJJ]

Be interesting to see if ImageIO.setUseCache(false) improves loading; or indeed, if avoiding ImageIO entirely is a better direction to go.

:edit:

A more in-depth & concise explanation.

- When ImageIO.setUseCache(true), ImageIO.read(InputStream) will wrap the supplied InputStream in a FileCacheImageInputStream. (caching read data to a temporary file on disk)
- When ImageIO.setUseCache(false), ImageIO.read(InputStream) will wrap the supplied InputStream in a MemoryCacheImageInputStream. (caching read data in memory)
- When ImageIO.read(ImageInputStream) is used, you can choose the caching strategy.
You can for instance pass in a FileImageInputStream, and no caching will be used at all (instead relying on the intrinsic random access support that Files have).

I'm not sure whether MemoryCacheImageInputStream, or FileImageInputStream will perform better; but I'd definitely expect the current behaviour (FileCacheImageInputStream) to be the poorest performing.

[Alex]

Hmm - looks like setting it to false makes the load time go from a bit over 8 seconds to a bit under 6. Will probably make more of a difference on a non-SSD. Thank you for pointing this out, wasn't aware that was how it worked.

[blaa]

*arise from the dead, thread*

I just wanted to give a little update on this. I have a new rig and it has improved my loading times significantly, however in comparison to other games the loading time is still pretty long.
With a similar mod selection as before I now reach the main menu in about 30-40 seconds. Without mods it´s only ~5-7 seconds. That´s 30+ seconds to load ~600 MB worth of mods.

Win10
I7 7700 @ 4.2Ghz
GTX 1080
32 GB Ram
Game on SSD

This isn´t a catastrophe but pretty much every game I play on steam boots up way faster, some of them tens of GB with many GB of mods.

[zaimoni]

Note that a very early prototype of a StarSector savefile scanner on my end (implemented in Ruby) needs ~93 seconds just to parse 25.7MB of XML.  I don't think Java will be astronomically faster than Ruby (they're both virtual machine languages and neither is specifically optimized for text processing), and C isn't markedly faster than Ruby either, so I just expect relatively slow game load times no matter what.

[Alex]

Glad it's improved, at any rate! And for the base game, it sounds like it's in a reasonable ballpark.

As far as with mods, I'm curious to see how disabling GraphicsLib from its settings file would change the loading time - iirc it loads a lot of normal maps to give the ship sprites some 3D-ness for the lighting effects, but I'm not sure how big a part of the total loading time it is.

This isn´t a catastrophe but pretty much every game I play on steam boots up way faster, some of them tens of GB with many GB of mods.

One thing Starsector gets here is a lack of loading screens once in-game. Also, I wouldn't necessarily compare filesize here - it *can* contribute to loading time, but it is also far from being the only factor, and just looking at file sizes is going to be very misleading.

Still, faster loading would be nice, of course.

Note that a very early prototype of a StarSector savefile scanner on my end (implemented in Ruby) needs ~93 seconds just to parse 25.7MB of XML.  I don't think Java will be astronomically faster than Ruby (they're both virtual machine languages and neither is specifically optimized for text processing), and C isn't markedly faster than Ruby either, so I just expect relatively slow game load times no matter what.

(That's a bit off-topic; let's please try and keep this subforum on track. That said, 93 seconds for just parsing XML - assuming you're not actually doing anything with it, but just reading the file, and assuming it's a reasonably-powered computer - seems way high. For a file that big, I'd suggest making sure you're using a streaming XML parsing API, and not a DOM-based one. The performance difference for larger files will be major. For reference, I've got a savefile scanner in Java - for analyzing how big certain elements are - and it takes something like a second for a 10MB file, including the actual analysis.)