Fractal Softworks Forum

Starsector => Suggestions => Topic started by: LucusLoC on January 18, 2020, 01:48:40 PM

Title: Save File Format (Game arcitecture)
Post by: LucusLoC on January 18, 2020, 01:48:40 PM
I am currently playing through a "full colonization" play-through, and this playstyle is really taxing the games ability to handle data. My save file is getting really close to 50 MB of XML, and when the game needs to access some random data (say colony info) things really chug as the file is parsed and the needed data is located. It occurs to me that we could probably get better performance by breaking save file down into discrete chunks so that only the needed data would need to be parsed, instead of the entire save file.

As an example structure:

The idea here is to reduce the "seek time" when the game is accessing various bits of data. In this example, when a player wants to bring up colony management info, instead of crawling through a 50MB XML file looking for the tags for that one colony, the game would only need  to crawl the "Sector main" file for the proper system ID, then crawl the appropriate "Market file" for the colony in question. This should reduce the data being parsed to a few MB at most, which should drastically increase the speed of accessing that data.

I would also recommend a move away from XML, but I understand that moving away from that format would make support and modding significantly harder. That wold probably go against one of the main paradigms of the game's design. (also, not being a Java programmer I am not 100% sure how much performance there is to gain from switching from XML to some sort of raw binary data. Maybe someone else can offer some insight there. Learning Java is on my to-do list, so feel free to delve into details.) I would hope that cutting down on file sizes would offer enough of a performance improvement to make such a drastic change unnecessary.

There is a downside to such a change, however, and that is (probably) increased save times. Instead of doing one file access and write operation the game would need to stage and execute multiple file writes for every save. To do the save properly ("robustly") the game will need to track the status of the various writes, and have some way to roll back in case of failure (or else risk corruption should some cross-file references go missing). Personally I find the trade-off of longer saves to be worth less lag while actually playing.

Let me know if anyone else things a save file architecture change would be worth the time and effort.
Title: Re: Save File Format (Game arcitecture)
Post by: LazyWizard on January 18, 2020, 02:31:13 PM
The entire save file is loaded from disk and deserialized into the proper game objects when you first load your save. The game doesn't otherwise read the save file during gameplay. The only disk I/O downtime you'll run into is the actual saving/loading screens, and the slight delay when you jump into a system as it loads the background texture.

AFAIK, the lag when opening up some screens is due to game objects being created to populate the UI. I know mousing over a fleet with a large number of ships can cause the game to stutter noticeably as it generates each ship's stats card. I assume the colony screen has the same problem.
Title: Re: Save File Format (Game arcitecture)
Post by: Alex on January 18, 2020, 02:50:01 PM
Yeah, any time you open the colony screen, the game force-updates the economy. If there's a ton of colonies (which "full colonization" kind of implies) it's going to chug at some point. Right now you can colonize way more stuff effectively (using Alpha Cores) than you'd ultimately be able to before running into problems, so having more than X* new colonies is basically "outside the intended scope" territory and performance suffering a bit there is to be expected.

*X is TBD, but likely under 10.
Title: Re: Save File Format (Game arcitecture)
Post by: LucusLoC on January 18, 2020, 07:00:55 PM
Ah, I see then. Basically, this style of gameplay is unintended (and I presume will be nerfed in some future patch). And the lag is not a read/parse issue, it is a "recalculate the entire game state" issue.

I guess I will wait and see what the endgame comes to, as this is the only real logical endgame I can see right now (vanilla or modded). I have "settled" inter-factional differences; I can farm [Redacted] on an as-needed basis, I have more money than I know what to do with. What is left to do but colonize the next system? And the next, and the next?

I mean, it would be nice if the finished game let this be an optional end game, but given the limitations mentioned that may be more trouble to optimize than it is worth. I will probably keep playing this save until the lag becomes unbearable, then find something else to do. Thanks for the replies.
Title: Re: Save File Format (Game arcitecture)
Post by: Alex on January 18, 2020, 07:05:26 PM
To be clear, I totally get where you're coming from, and applaud your doing it :)
Title: Re: Save File Format (Game arcitecture)
Post by: LucusLoC on January 18, 2020, 08:59:54 PM
Thanks for making a game that is entertaining enough to play on such a level. It is just a shame that so many great game eventually run into this scaleability issue. I killed most of my Dwarf Fortress or Kerbal Space Program games doing a similar thing; grow beyond the capabilities of the game to handle it. Maybe someday when the game is more or less feature complete you can do a full release cycle of just optimization, but that is a long ways off.

Do let me know if you need someone to do, ah, "stress testing" on any release candidate version ;-)
Title: Re: Save File Format (Game arcitecture)
Post by: Alex on January 19, 2020, 10:45:10 AM
Thanks for making a game that is entertaining enough to play on such a level. It is just a shame that so many great game eventually run into this scaleability issue. I killed most of my Dwarf Fortress or Kerbal Space Program games doing a similar thing; grow beyond the capabilities of the game to handle it. Maybe someday when the game is more or less feature complete you can do a full release cycle of just optimization, but that is a long ways off.

If you think about it, that's actually unavoidable! What to spend dev time on is always a tradeoff, right - it's time not spent on something else - so you always want to spend it efficiently. Trying to optimize for a case that ultimately doesn't matter very much is counter-productive - that time would be better spent, say, adding more content, or even just optimizing something else that matters more.
Title: Re: Save File Format (Game arcitecture)
Post by: SCC on January 19, 2020, 12:24:44 PM
Hey Alex, was there a significant change between 0.9 and 0.9.1 in this regard? I remember my 313 colony run having some memory leak issues, but accessing colonies didn't feel much different.
Title: Re: Save File Format (Game arcitecture)
Post by: Alex on January 19, 2020, 09:24:58 PM
Nothing that comes to mind, no. It was definitely a bit slower to open the screen, though, even in 0.9 - not, like, seconds more, but enough that it was noticeable for me. The 50mb savefile here sounds like some kind of (probably?) mod-related memory leak, though, which could also mean that some of the slowdown is due to memory use being too high for the allocation, unless vmparams was tweaked. I'm not 100% sure, of course, since I don't even know if the game in question is vanilla or not - just, 50mb sounds a bit high to happen "naturally".