I am currently playing through a "full colonization" play-through, and this playstyle is really taxing the games ability to handle data. My save file is getting really close to 50 MB of XML, and when the game needs to access some random data (say colony info) things really chug as the file is parsed and the needed data is located. It occurs to me that we could probably get better performance by breaking save file down into discrete chunks so that only the needed data would need to be parsed, instead of the entire save file.
As an example structure:
- Sector main: raw sector info. IDs for systems, planets and other "natural" features.
- Character main: Money, Fleet info (ships and inventory), level info.
- Sector Activity main: Info on all the various fleets moving about, individual entities not attached to markets, various bits of intel, faction data, quest data and the status of bits of loot floating around.
- Market files <system ID#>: individual save files for each system's markets, as well as "system ownership" status. Could also include things like Remnant stations, occupied stable points, and other miscellaneous info that is dynamic within the system.
- Other main: A dumping ground for unclassified data that needs to be remembered, but has no other good place to be kept (i.e. does not need to be accessed at the same time as other data). Would also be a good place to store mod data that does not fit neatly into the other categories.
The idea here is to reduce the "seek time" when the game is accessing various bits of data. In this example, when a player wants to bring up colony management info, instead of crawling through a 50MB XML file looking for the tags for that one colony, the game would only need to crawl the "Sector main" file for the proper system ID, then crawl the appropriate "Market file" for the colony in question. This should reduce the data being parsed to a few MB at most, which should drastically increase the speed of accessing that data.
I would also recommend a move away from XML, but I understand that moving away from that format would make support and modding significantly harder. That wold probably go against one of the main paradigms of the game's design. (also, not being a Java programmer I am not 100% sure how much performance there is to gain from switching from XML to some sort of raw binary data. Maybe someone else can offer some insight there. Learning Java is on my to-do list, so feel free to delve into details.) I would hope that cutting down on file sizes would offer enough of a performance improvement to make such a drastic change unnecessary.
There is a downside to such a change, however, and that is (probably) increased save times. Instead of doing one file access and write operation the game would need to stage and execute multiple file writes for every save. To do the save properly ("robustly") the game will need to track the status of the various writes, and have some way to roll back in case of failure (or else risk corruption should some cross-file references go missing). Personally I find the trade-off of longer saves to be worth less lag while actually playing.
Let me know if anyone else things a save file architecture change would be worth the time and effort.
