Fractal Softworks Forum

Starsector => Bug Reports & Support => Bug Reports & Support (modded) => Topic started by: erikem on June 21, 2019, 11:43:52 AM

Title: Quicker save/load? (Save bloat bug? -Alex)
Post by: erikem on June 21, 2019, 11:43:52 AM
Hi,

I play a game with Nexelerin and bunch of factions, big galaxy (or is it called standard?). Lotsa factions but only a few select have more than 2 markets. My problem is that right now my compressed save file is already 61mb and save/load process takes uncomfortable amount of time even on SSD.

Is it expected? Is there any way to trim down the save file size or make saving/loading faster?
Title: Re: Quicker save/load?
Post by: SCC on June 21, 2019, 11:47:19 AM
Yes. Also yes, play with fewer mods. Mind you, 0.9 already saw major loading optimisation, so it could have been worse. You just have to live with it.
Title: Re: Quicker save/load?
Post by: erikem on June 21, 2019, 01:18:16 PM
And if my make "small" sector? Will it help or is it going to be the same in the long run?

Also how does "having less mods" actually help? Isn't the size of the save file driven by the number of active markets and fleets? I mean if I have 10 factions x2 markets each, x10 fleets each (20 markets and 100 fleets in total) will it be different from 2 factions with 5 markets and 50 fleets each?
Title: Re: Quicker save/load?
Post by: Alex on June 21, 2019, 08:06:45 PM
Hmm, this kind of makes me wonder whether mods are taking advantage of the "routemanager" system for inflating/deflating fleets and so on. 60mb seems like surprisingly a lot; I'm not sure what would take up that much space. Something seems off about that.

My problem is that right now my compressed save file is already 61mb

Just to confirm, you've got savefile compression turned on via settings.json? It might actually be faster with that turned off, especially if you've got an SSD, since disk writes are cheap and compressing a large savefile is less so.
Title: Re: Quicker save/load?
Post by: erikem on June 21, 2019, 08:43:55 PM
Just to confirm, you've got savefile compression turned on via settings.json? It might actually be faster with that turned off, especially if you've got an SSD, since disk writes are cheap and compressing a large savefile is less so.
Yup, save file compression is turned on via settings.json:
"compressSaveGameData":true

Uncompressed campaign.xml is like 277mb. Actually the speed seems to be the same both with and without compression and probably that's coz of otherwise fast PC.

I also have kind of a vague feeling that save file went almost twice as large when I started exploring the unknown but that could simply be a coincidence.

I do "save copy" quite often and here is the dynamics (please disregard the top save with 399mb size; compression was enabled since save_ErikemII_7906960025789190898; some of the saves might look suspiciously small because I never did the second save on that file (did "copy save" and next time did "copy save" again) and thus they don't have *.baks):
(https://i.imgur.com/gVHXBIf.png)
Title: Re: Quicker save/load?
Post by: Midnight Kitsune on June 21, 2019, 08:49:26 PM
Holy S***! I've NEVER seen save files that big! I THINK I've heard of a mod causing save bloat issues (Vayra's sector I THINK). But yeah, I'm pretty sure the shear sizes of those saves is what is taking you so long to save your game
Title: Re: Quicker save/load?
Post by: Alex on June 21, 2019, 09:06:02 PM
... oh, wow! Yeah, this is definitely not normal-looking; as Midnight says, this looks like it's likely a save bloat bug in one of the mods you're using.
Title: Re: Quicker save/load?
Post by: erikem on June 21, 2019, 09:12:39 PM
Just to be clear these sizes are from the saves before I turned on the save file compression. Here are the save after I turned on the compression:
(https://i.imgur.com/r6Wa6AL.png)

Also some of the saves might look suspiciously small because I never did the second save on that file (did "copy save" and next time did "copy save" again) and thus they don't have *.baks

Holy S***! I've NEVER seen save files that big! I THINK I've heard of a mod causing save bloat issues (Vayra's sector I THINK). But yeah, I'm pretty sure the shear sizes of those saves is what is taking you so long to save your game
Yup... I kind of understand that the problem is in the sheer size =( And yes I do have "Vayra's sector". Are there any other mods infamous for save bloating?

Here's the list of mods I've been using:
Spoiler
{"enabledMods": [
  "$$$_trailermoments",
  "anotherportraitpack",
  "raccoonarms",
  "blackrock_driveyards",
  "CombatAnalytics",
  "chatter",
  "lw_radar",
  "istl_dam",
  "diableavionics",
  "DisassembleReassemble",
  "gs",
  "GMDA",
  "HMI",
  "HA",
  "junk_pirates_release",
  "lw_lazylib",
  "leadingPip",
  "ArkLeg",
  "luddenhance",
  "MagicLib",
  "Mayasuran Navy",
  "nexerelin",
  "ORA",
  "sun_ruthless_sector",
  "sad",
  "SCY",
  "shadow_ships",
  "swp",
  "bonomel_skilledup",
  "speedUp",
  "Sylphon_RnD",
  "tahlan",
  "underworld",
  "US",
  "URW",
  "vayrasector",
  "lw_version_checker",
  "audio_plus",
  "shaderLib"
]}
[close]
Title: Re: Quicker save/load?
Post by: erikem on June 21, 2019, 09:14:50 PM
... oh, wow! Yeah, this is definitely not normal-looking; as Midnight says, this looks like it's likely a save bloat bug in one of the mods you're using.
Any ideas how can I identify the mods causing save bloats? Apart from playtesting various mod combinations which could take like eternity =)
Title: Re: Quicker save/load?
Post by: Alex on June 21, 2019, 09:19:59 PM
Any ideas how can I identify the mods causing save bloats? Apart from playtesting various mod combinations which could take like eternity =)

Well, you could try to open up the xml in some kind of editor that can handle files this big, and see if anything jumps out as taking up a lot of the file, but that could be pretty tricky to figure out.

Beyond that, yeah, testing with combinations of mods would be the way to go. Probably fastest to do half the mods (aside from the utility mods) at a time, so that you can tell which half of the mods the problem is in, and then progressively narrow it down from there. Still quite a bit of time/effort, though.

... I THINK I've heard of a mod causing save bloat issues (Vayra's sector I THINK). ...

And that might be a good one to start with as just a one-off...
Title: Re: Quicker save/load?
Post by: erikem on June 21, 2019, 09:29:01 PM
Well, you could try to open up the xml in some kind of editor that can handle files this big, and see if anything jumps out as taking up a lot of the file, but that could be pretty tricky to figure out.
Notepad++ can handle it but I'm not sure I'll be able to find the issue. It's not like each mode is putting it's own stuff in a separate big block

Beyond that, yeah, testing with combinations of mods would be the way to go. Probably fastest to do half the mods (aside from the utility mods) at a time, so that you can tell which half of the mods the problem is in, and then progressively narrow it down from there. Still quite a bit of time/effort, though.
Yeah going halves each time is the fastest way but it took me liek 7 days to bloat it this size on single playthrough... spending another 5x7daays cycles just to track bloating mods (would be progressively harder if there are several of them) would just suck any fun of the game

... I THINK I've heard of a mod causing save bloat issues (Vayra's sector I THINK). ...

And that might be a good one to start with as just a one-off...
Thx! On my next playthrough I'll try without that mod coz save bloating is a pain. Also just by looking at the list of mods I have enabled - do you see any other suspicious ones?
Title: Re: Quicker save/load?
Post by: Alex on June 21, 2019, 09:33:05 PM
Notepad++ can handle it but I'm not sure I'll be able to find the issue. It's not like each mode is putting it's own stuff in a separate big block
...
Yeah going halves each time is the fastest way but it took me liek 7 days to bloat it this size on single playthrough... spending another 5x7daays cycles just to track bloating mods (would be progressively harder if there are several of them) would just suck any fun of the game

Yeah, not saying it's a good idea by any means, just nothing more effective comes to mind.

Thx! On my next playthrough I'll try without that mod coz save bloating is a pain. Also just by looking at the list of mods I have enabled - do you see any other suspicious ones?

I don't, but this is also the first time I'm hearing about the issue. Others are going to be more aware of the latest issues etc.

(... let me actually move this thread to bug reports (modded); that feels like where it belongs.)

Edit: also modified the thread title in the OP.
Title: Re: Quicker save/load? (Save bloat bug? -Alex)
Post by: Vayra on June 21, 2019, 10:09:08 PM
Yeah, this is probably me. Could you upload one of the larger saves for me? I'm not sure where to start diagnosing something like this.
Title: Re: Quicker save/load? (Save bloat bug? -Alex)
Post by: erikem on June 21, 2019, 10:27:21 PM
Yeah, this is probably me. Could you upload one of the larger saves for me? I'm not sure where to start diagnosing something like this.
Sure I will. I have just checked and thx to the tidy way you pack stuff in XML I was able to locate ur stuff and check the size. Out of the 270mb of campaign.xml 247mb are indeed taken by your mod.

Here's the campaign.xml I was analyzing: https://www.dropbox.com/s/vsqbpjfaa8hmchx/campaign.xml?dl=0

In order to be able to navigate an XML of this size with ease I was using: http://www.firstobject.com/dn_editor.htm (much faster than Notepad++ on files of this size)

Please have a look at this section of XML (VayraRaiderBaseReaper/hiddenMarkets/ is like the block that does all the bloating coz this block alone is 247mb while the rest of your blocks are less than 1mb):
/CampaignEngine/data.scripts.campaign.bases.VayraRaiderBaseReaper/hiddenMarkets/Market[]/economy/stepper/econ/markets/Market/primaryEntity/orbit/f/where/o/saved - seems all the bloating happens here
Title: Re: Quicker save/load? (Save bloat bug? -Alex)
Post by: Vayra on June 21, 2019, 10:36:07 PM
Yeah, this is probably me. Could you upload one of the larger saves for me? I'm not sure where to start diagnosing something like this.
Sure I will. I have just checked and thx to the tidy way you pack stuff in XML I was able to locate ur stuff and check the size. Out of the 270mb of campaign.xml 247mb are indeed taken by your mod.

Here's the campaign.xml I was analyzing: https://www.dropbox.com/s/vsqbpjfaa8hmchx/campaign.xml?dl=0

In order to be able to navigate an XML of this size with ease I was using: http://www.firstobject.com/dn_editor.htm (much faster than Notepad++ on files of this size)

Please have a look at this section of XML (VayraRaiderBaseReaper/hiddenMarkets/ is like the block that does all the bloating coz this block alone is 247mb while the rest of your blocks are less than 1mb):
/CampaignEngine/data.scripts.campaign.bases.VayraRaiderBaseReaper/hiddenMarkets/Market[]/economy/stepper/econ/markets/Market/primaryEntity/orbit/f/where/o/saved - seems all the bloating happens here

Thanks for the detailed info! This helps a ton.
Title: Re: Quicker save/load? (Save bloat bug? -Alex)
Post by: Vayra on June 21, 2019, 10:44:13 PM
Yep, looks like I was probably breaking garbage collection somehow -- My only java experience is via SS modding, so I'm at a bit of a loss when it comes to stuff like this. Thanks to you identifying the culprit I'll poke at it and hopefully have a fix later tonight.
Title: Re: Quicker save/load? (Save bloat bug? -Alex)
Post by: erikem on June 22, 2019, 01:09:38 AM
You are welcome and hoping you'll be able to fix it soon.
Title: Re: Quicker save/load? (Save bloat bug? -Alex)
Post by: Alex on June 22, 2019, 10:09:56 AM
So... I investigated the savefile, and it looks like about 150mb of that is just asteroids, looking like so:

Code
<Astrd z="324797" rot="-1.7" r="9.6">
<l z="324798">-7831.8853|7521.065</l>
<v z="324799">0.0|0.0</v>
<j0>{"f6":-965.7,"f0":"Asteroid","f2":[255,203,186,255],"f4":"abf0a53f-4c8e-48a0-a3ce-a1fbdff118cd"}</j0>
<o ref="173963"></o>
<orbit cl="COrbt" z="324800" r="10858.399" op="959.9271" ca="136.15979">
<e cl="Astrd" ref="324797"></e>
<f cl="CampaignTerrain" ref="324710"></f>
</orbit>
<cL cl="Sstm" ref="324508"></cL>
<ls cl="Plnt" ref="324516"></ls>
<me z="324801">
<d z="324802"></d>
</me>
<sc z="324803"></sc>
<cD z="324804">
<e>
<st>misc_astrdSource</st>
<AsteroidBeltTerrainPlugin ref="324716"></AsteroidBeltTerrainPlugin>
</e>
</cD>
</Astrd>

There are almost 250,000 of these in the save. To clarify, this is data for the individual, clickable-on asteroids that are part of asteroid fields or belts.

What the game normally does is not save this data at all (unless the asteroid's memory has something in it), and instead re-generates the asteroids on game load. The only case where the game will normally save asteroids - when the asteroid's memory has something in it, indicating something cares about the individual asteroid - it will also remove this data from it:

<st>misc_astrdSource</st>
<AsteroidBeltTerrainPlugin ref="324716"></AsteroidBeltTerrainPlugin>

So, it looks like the code for the game to remove asteroids from savefiles is being bypassed entirely. This code is in CoreLifecyclePluginImpl.beforeGameSave().

The other thing this code does is remove orbital junk from around populated/ruined worlds; this is also regenerated on game load. The orbital junk also appears in the savefile (there are 22,000 or so of these in there); this supports the hypothesis that the code that cleans this up is not running at all.

The most likely culprit, then seems like a mod that specifies a "coreLifecyclePlugin" in its settings.json, with an implementation that does not do these save-optimization tasks that vanilla does. I don't know which mod this would be.


In order to be able to navigate an XML of this size with ease I was using: http://www.firstobject.com/dn_editor.htm (much faster than Notepad++ on files of this size)

Nice! Downloaded this and it was pretty handy; UltraEdit was struggling a bit with this.

Please have a look at this section of XML (VayraRaiderBaseReaper/hiddenMarkets/ is like the block that does all the bloating coz this block alone is 247mb while the rest of your blocks are less than 1mb):
/CampaignEngine/data.scripts.campaign.bases.VayraRaiderBaseReaper/hiddenMarkets/Market[]/economy/stepper/econ/markets/Market/primaryEntity/orbit/f/where/o/saved - seems all the bloating happens here

Just for general information - the way savefiles are structured, just because something is inside another element doesn't mean very much. The save code picks some "root" to start from and just goes from there, so for example a fleet might contain a reference to the star system it's in, which might contain a reference to the entire Sector. So basically the nesting order really doesn't mean much. I hope that makes some sort of sense :)
Title: Re: Quicker save/load? (Save bloat bug? -Alex)
Post by: erikem on June 24, 2019, 11:08:34 AM
Thank you very much Alex. Hope this info will be useful for Vayra.

PS indeed XML structure looks strange for those unprepared. Thx for explaining it
Title: Re: Quicker save/load? (Save bloat bug? -Alex)
Post by: Alex on June 24, 2019, 11:13:45 AM
Thank you very much Alex. Hope this info will be useful for Vayra.

What I'm saying is it might not be Vayra's mod causing the issue :) Just the XML nesting happens to make it look like it might be but that doesn't count for too much. But it still *could* be Vayra's mod. Whoever's providing that core plugin override, basically.
Title: Re: Quicker save/load? (Save bloat bug? -Alex)
Post by: Vayra on June 24, 2019, 12:15:02 PM
Ah, hmmm. Yeah, I don't provide one of those.  :-X
Title: Re: Quicker save/load? (Save bloat bug? -Alex)
Post by: Histidine on June 24, 2019, 05:03:05 PM
Nex has a core lifecycle plugin replacement, but it extends the vanilla one and only overrides addScriptsIfNeeded. I haven't seen anything like this issue in my own Nex games and haven't received any such reports.

Don't know which other mods in circulation replace the plugin.
Title: Re: Quicker save/load? (Save bloat bug? -Alex)
Post by: Alex on June 24, 2019, 05:42:16 PM
Nex has a core lifecycle plugin replacement, but it extends the vanilla one and only overrides addScriptsIfNeeded. I haven't seen anything like this issue in my own Nex games and haven't received any such reports.

That sounds like it should be totally fine, yeah.

Whatever plugin it is, it most likely overrides both beforeGameSave() (since that's where the asteroids are removed) and onGameLoad() (since if that wasn't overridden, more and more asteroids would get added to the game with each save/load...)
Title: Re: Quicker save/load? (Save bloat bug? -Alex)
Post by: Vayra on June 25, 2019, 04:33:04 PM
Did a quick search through all the mods posted in that modlist with Notepad++'s "find in files" and the only mod that has a coreLifecyclePlugin in its settings.json is Nexerelin, but like Histidine said, that one should work as it extends CoreLifecyclePluginImpl and the only method in it is addScriptsIfNeeded()... Hmm.

Wait, that should work, right? No chance the game is replacing CoreLifecyclePluginImpl and then not calling the parent/super's onGameLoad because it's not explicitly included in the extension that replaced it?

Because if that's not it, I'm stumped. Maybe if something silently crashes beforeGameSave and onGameLoad?

----

Alternatively, reading through this .xml it looks like my hiddenMarkets list sometimes catches just a "ref", whatever that is, literally just a number -- and sometimes catches a "z" with all the details on a market plus, apparently, all of the campaign terrain, inside another market inside an "economy stepper".

Path looks like this (line broken for readability):
/CampaignEngine/scripts/data.scripts.campaign.bases.VayraRaiderBaseReaper/hiddenMarkets/Market[8]/economy/stepper/
econ/markets/Market/primaryEntity/orbit/f/where/o/saved/
LocationToken[147]/orbit/s/con/systems/Sstm[5]/o/saved/
Astrd[196]/cD/e/AsteroidBeltTerrainPlugin

Another, longer one, lives under a different market:
/CampaignEngine/scripts/data.scripts.campaign.bases.VayraRaiderBaseReaper/hiddenMarkets/Market[25]/economy/stepper/
econ/markets/Market/primaryEntity/orbit/f/where/o/saved/
LocationToken/orbit/s/con/systems/Sstm[4]/o/saved/
Plnt[2]/market/submarkets/Submarket[4]/f/production/gatheringPoint/admin/stats/fleet/cL/o/saved/
Flt[3]/ai/n/a/items/IDt[2]/i/e/dL/RouteManager/r/RouteData[33]/e/RtSeg[2]/t/cL/con/systems/Sstm[3]/o/saved/
Astrd[252]/cD/e/AsteroidBeltTerrainPlugin

So maybe my fixing that list will fix this? I'm gonna try it (actually already have), so fingers crossed?   :-X
Title: Re: Quicker save/load? (Save bloat bug? -Alex)
Post by: Alex on June 25, 2019, 05:34:24 PM
Did a quick search through all the mods posted in that modlist with Notepad++'s "find in files" and the only mod that has a coreLifecyclePlugin in its settings.json is Nexerelin, but like Histidine said, that one should work as it extends CoreLifecyclePluginImpl and the only method in it is addScriptsIfNeeded()... Hmm.

Wait, that should work, right? No chance the game is replacing CoreLifecyclePluginImpl and then not calling the parent/super's onGameLoad because it's not explicitly included in the extension that replaced it?

Because if that's not it, I'm stumped. Maybe if something silently crashes beforeGameSave and onGameLoad?

No, that should be fine. For any method not included in the extending class, the version in the parent class will be used - that's kind of a core thing about how inheritance works. And it's tough to see how beforeGameSave() could fail silently.

So, that's weird. A couple of things point towards CoreLifecyclePluginImpl.beforeGameSave() not being called for some reason - if it was just asteroids, that would be doubtful. But it's both asteroids and orbital_junk, the two things that method clears out.

Alternatively, reading through this .xml it looks like my hiddenMarkets list sometimes catches just a "ref", whatever that is, literally just a number -- and sometimes catches a "z" with all the details on a market plus, apparently, all of the campaign terrain, inside another market inside an "economy stepper".

Path looks like this (line broken for readability):
/CampaignEngine/scripts/data.scripts.campaign.bases.VayraRaiderBaseReaper/hiddenMarkets/Market[8]/economy/stepper/
econ/markets/Market/primaryEntity/orbit/f/where/o/saved/
LocationToken[147]/orbit/s/con/systems/Sstm[5]/o/saved/
Astrd[196]/cD/e/AsteroidBeltTerrainPlugin

That path basically doesn't matter and is not an indication of WHY something is in the savefile or what it relates to. It'll be super misleading if you treat it as such.

"ref" is a reference to another element that's found in full detail elsewhere in the savefile. For example, a market's data will be written out in full once, and then whenever something else refers to that market, it'll use "ref=<that market's savefile ID>"

"z" is the attribute which contains that unique (to the savefile, and only relevant to that specific savefile) id.
Title: Re: Quicker save/load? (Save bloat bug? -Alex)
Post by: Ishman on June 25, 2019, 07:32:37 PM
Did a fast run with the only non utility mod being vayra's sector, completed one pirate bounty and dipped into four star systems total. Ctrl f the save and found 2000 instances of asteroids.

http://www.mediafire.com/file/7iihyaruabfad8p/campaign.xml/file
Title: Re: Quicker save/load? (Save bloat bug? -Alex)
Post by: Alex on June 25, 2019, 08:09:11 PM
Thank you for giving this a try! Checked the file, it actually looks fine - the specific things that you'd want to look for is "Astrd" and "orbital_junk". Looks like neither is ending up in the save, at least at this point. The file size is also fine - around 10mb is about what I'd expect here.

(The occurrences of "asteroid" are legitimate, i.e. things like the terrain rather than individual asteroids; texture names, etc.)
Title: Re: Quicker save/load? (Save bloat bug? -Alex)
Post by: Ishman on June 25, 2019, 08:14:48 PM
Gotcha, going to go through all the released mods to find the cause then - it should've already showed up since I did dip through a system with asteroid entities.

Edit: So far I believe I've ruled out BRDY, Interstellar Imperium, Arsenal Expansion, and Vayra's sector.

Edit2: Further info - the problem does not present itself quickly, it seems, I went through enabling one by one half of everything I've got - then enabled everything and did the same (Add fuels and supplies, then fly around between systems with asteroids, saving in hyperspace, in system, in hyperspace again, reloading every so often after a save). It doesn't seem to immediately present itself, so it might be something that you have to leave the game idling for a long time to start noticing, which will make it much harder to narrow down. It didn't seem to become apparent when spamming save/load/save copy.
Title: Re: Quicker save/load? (Save bloat bug? -Alex)
Post by: Alex on June 26, 2019, 08:51:33 AM
This *seems* like something that would manifest immediately, and not even require flying around the various systems. Unless I'm wrong about the cause, of course, which is possible.

So the fact that you're been unable to reproduce it seems to suggest that the set of mods you're using doesn't cause the problem at all.

Just to confirm, you've tried every mod from this list?
Here's the list of mods I've been using:
Spoiler
{"enabledMods": [
  "$$$_trailermoments",
  "anotherportraitpack",
  "raccoonarms",
  "blackrock_driveyards",
  "CombatAnalytics",
  "chatter",
  "lw_radar",
  "istl_dam",
  "diableavionics",
  "DisassembleReassemble",
  "gs",
  "GMDA",
  "HMI",
  "HA",
  "junk_pirates_release",
  "lw_lazylib",
  "leadingPip",
  "ArkLeg",
  "luddenhance",
  "MagicLib",
  "Mayasuran Navy",
  "nexerelin",
  "ORA",
  "sun_ruthless_sector",
  "sad",
  "SCY",
  "shadow_ships",
  "swp",
  "bonomel_skilledup",
  "speedUp",
  "Sylphon_RnD",
  "tahlan",
  "underworld",
  "US",
  "URW",
  "vayrasector",
  "lw_version_checker",
  "audio_plus",
  "shaderLib"
]}
[close]

@erikem: From your experience, did the saves start out relatively large? E.G. say something larger than 20-30mb uncompressed?

Edit: If you happen to have a savefile where this is reproducible (both the campaign and the descriptor xml), I'd love to get my hands on it! And if you would also zip up and upload your mod folder somewhere, that would make my life a lot easier, though I know that's a bit to ask :)
Title: Re: Quicker save/load? (Save bloat bug? -Alex)
Post by: CynicJester on June 26, 2019, 11:32:36 AM
I'm not sure if this is related or not, but I was having issues saving my game until I removed "luddenhance". I seem to recall what save files I got being bloated prior to removing it, but I already deleted them, so I can't say with complete certainty that it's related.

Edit: It seems unrelated, as upon checking my current save without the mod, there are some 163k instances of the word "Astrd", which seems excessive if they're supposed to be pruned.

Current modlist:

Spoiler
{"enabledMods": [
  "$$$_lightshow",
  "$$$_trailermoments",
  "lw_autosave",
  "blackrock_driveyards",
  "lw_radar",
  "lw_lazylib",
  "leadingPip",
  "MagicLib",
  "nexerelin",
  "ogaportraitpack",
  "shadow_ships",
  "swp",
  "bonomel_skilledup",
  "transponder-reminders",
  "URW",
  "vayrasector",
  "lw_version_checker",
  "audio_plus",
  "shaderLib",
  "Safety Override mod"
]}
[close]
Title: Re: Quicker save/load? (Save bloat bug? -Alex)
Post by: Alex on June 26, 2019, 12:04:05 PM
Thank you for chiming in! Any chance you could send me your save? And if you're feeling particularly generous, your mod folder as well? :)

Would probably have to upload those somewhere since I suspect the save would also be too large for an email attachment.
Title: Re: Quicker save/load? (Save bloat bug? -Alex)
Post by: CynicJester on June 26, 2019, 12:39:49 PM
Thank you for chiming in! Any chance you could send me your save? And if you're feeling particularly generous, your mod folder as well? :)

Would probably have to upload those somewhere since I suspect the save would also be too large for an email attachment.

https://drive.google.com/open?id=1g7YzdZTaFLwlZuEFVETZYUyrq2BHuoNf

https://drive.google.com/open?id=1gl7TI7W_mlzfdLqZpRvEwKe3P-F3aePQ

Try those and see if they work. I haven't actually used Google Drive for anything beyond sharing files with my phone, so it might make something blow up.
Title: Re: Quicker save/load? (Save bloat bug? -Alex)
Post by: Alex on June 26, 2019, 02:35:40 PM
Thank you so much!

Dug into this, and looks like it's VayraSector causing the issues. Turning it off should help, unless something else is *also* causing the same exact problem. Which, I mean, unlikely but possible.


Technical things below:
-----------------------------------------------------------------------------------------------
In a few of its classes there are things like:

private static final SectorAPI SECTOR = Global.getSector();
private static final SectorAPI sector = Global.getSector();

Because the variable is static, it means that the sector stored in that variable will remain the same when a savegame is reloaded, so it'll be pointing to the old sector. This is by itself a memory leak, though static fields don't get saved and so this would not affect the savefile directly.

However, what I think is happening is this old, no-longer-valid Sector is being referenced in these classes, and some references being saved in non-static fields. These fields *do* end up in the save file, and pull the rest of the old sector in with them.

Since the beforeGameSave() code only goes through the current sector - it has no knowledge of any old instances being kept around - the old sector's asteroids aren't cleaned up and end up in the save. Along with the rest of the sector in that save, so there's duplicates of everything. For example, searching for "chicomoztoc" in the save reveals two separate versions of the market - one "real", one from the invalid sector.

As far as fixing this: for the love of Ludd, don't put ANYTHING AT ALL (aside from primitives) into static variables unless you really know what you're doing. Ahem.
Title: Re: Quicker save/load? (Save bloat bug? -Alex)
Post by: Vayra on June 26, 2019, 03:04:40 PM
Ah, miracles! Okay, I can fix that pretty easily then I imagine. Thanks! Expect a mod update later today.  ;D
Title: Re: Quicker save/load? (Save bloat bug? -Alex)
Post by: Vocation on July 19, 2020, 12:41:24 AM
I have the same issue as this. It starts when memory gets around 30% for me. I allocated 7.7gb to the game. I know because you can check it with console command mod.
Title: Re: Quicker save/load? (Save bloat bug? -Alex)
Post by: Zephroze on March 24, 2022, 11:33:21 PM
Sorry for the necropost but considering this is already similar to my line of questioning and eliminates some potential bad actors in the code, I think it's the most appropriate place.

I too am having an issue but it's not save bloat. It's not memory bloat either as far as I know.

The longer I play the game within one session, the longer it takes to save the game. Within under an hour it can take up to 30 seconds to save the game, with it only getting worse the longer I play. Then I have to restart the game which takes a couple minutes. Pretty big mod list.

I've allocated 6gb of memory and I have 16gb total, but it's only using around 3.7gb at most before it starts chugging like this.

Any ideas what I can do? I can post saves and logs and such if somebody is willing to help. I'll also take suggestions on a mod manager that lets me change the load order, seeing as I'm pretty competent with managing mod lists if there's a tool that can do it.