Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Vayra]

Yep, looks like I was probably breaking garbage collection somehow -- My only java experience is via SS modding, so I'm at a bit of a loss when it comes to stuff like this. Thanks to you identifying the culprit I'll poke at it and hopefully have a fix later tonight.

[erikem]

You are welcome and hoping you'll be able to fix it soon.

[Alex]

So... I investigated the savefile, and it looks like about 150mb of that is just asteroids, looking like so:

Code

<Astrd z="324797" rot="-1.7" r="9.6">
<l z="324798">-7831.8853|7521.065</l>
<v z="324799">0.0|0.0</v>
<j0>{"f6":-965.7,"f0":"Asteroid","f2":[255,203,186,255],"f4":"abf0a53f-4c8e-48a0-a3ce-a1fbdff118cd"}</j0>
<o ref="173963"></o>
<orbit cl="COrbt" z="324800" r="10858.399" op="959.9271" ca="136.15979">
<e cl="Astrd" ref="324797"></e>
<f cl="CampaignTerrain" ref="324710"></f>
</orbit>
<cL cl="Sstm" ref="324508"></cL>
<ls cl="Plnt" ref="324516"></ls>
<me z="324801">
<d z="324802"></d>
</me>
<sc z="324803"></sc>
<cD z="324804">
<e>
<st>misc_astrdSource</st>
<AsteroidBeltTerrainPlugin ref="324716"></AsteroidBeltTerrainPlugin>
</e>
</cD>
</Astrd>

There are almost 250,000 of these in the save. To clarify, this is data for the individual, clickable-on asteroids that are part of asteroid fields or belts.

What the game normally does is not save this data at all (unless the asteroid's memory has something in it), and instead re-generates the asteroids on game load. The only case where the game will normally save asteroids - when the asteroid's memory has something in it, indicating something cares about the individual asteroid - it will also remove this data from it:

<st>misc_astrdSource</st>
<AsteroidBeltTerrainPlugin ref="324716"></AsteroidBeltTerrainPlugin>

So, it looks like the code for the game to remove asteroids from savefiles is being bypassed entirely. This code is in CoreLifecyclePluginImpl.beforeGameSave().

The other thing this code does is remove orbital junk from around populated/ruined worlds; this is also regenerated on game load. The orbital junk also appears in the savefile (there are 22,000 or so of these in there); this supports the hypothesis that the code that cleans this up is not running at all.

The most likely culprit, then seems like a mod that specifies a "coreLifecyclePlugin" in its settings.json, with an implementation that does not do these save-optimization tasks that vanilla does. I don't know which mod this would be.

In order to be able to navigate an XML of this size with ease I was using: http://www.firstobject.com/dn_editor.htm (much faster than Notepad++ on files of this size)

Nice! Downloaded this and it was pretty handy; UltraEdit was struggling a bit with this.

Please have a look at this section of XML (VayraRaiderBaseReaper/hiddenMarkets/ is like the block that does all the bloating coz this block alone is 247mb while the rest of your blocks are less than 1mb):
/CampaignEngine/data.scripts.campaign.bases.VayraRaiderBaseReaper/hiddenMarkets/Market[]/economy/stepper/econ/markets/Market/primaryEntity/orbit/f/where/o/saved - seems all the bloating happens here

Just for general information - the way savefiles are structured, just because something is inside another element doesn't mean very much. The save code picks some "root" to start from and just goes from there, so for example a fleet might contain a reference to the star system it's in, which might contain a reference to the entire Sector. So basically the nesting order really doesn't mean much. I hope that makes some sort of sense

[erikem]

Thank you very much Alex. Hope this info will be useful for Vayra.

PS indeed XML structure looks strange for those unprepared. Thx for explaining it

[Alex]

Thank you very much Alex. Hope this info will be useful for Vayra.

What I'm saying is it might not be Vayra's mod causing the issue Just the XML nesting happens to make it look like it might be but that doesn't count for too much. But it still *could* be Vayra's mod. Whoever's providing that core plugin override, basically.

[Vayra]

Ah, hmmm. Yeah, I don't provide one of those. 

[Histidine]

Nex has a core lifecycle plugin replacement, but it extends the vanilla one and only overrides addScriptsIfNeeded. I haven't seen anything like this issue in my own Nex games and haven't received any such reports.

Don't know which other mods in circulation replace the plugin.

[Alex]

Nex has a core lifecycle plugin replacement, but it extends the vanilla one and only overrides addScriptsIfNeeded. I haven't seen anything like this issue in my own Nex games and haven't received any such reports.

That sounds like it should be totally fine, yeah.

Whatever plugin it is, it most likely overrides both beforeGameSave() (since that's where the asteroids are removed) and onGameLoad() (since if that wasn't overridden, more and more asteroids would get added to the game with each save/load...)

[Vayra]

Did a quick search through all the mods posted in that modlist with Notepad++'s "find in files" and the only mod that has a coreLifecyclePlugin in its settings.json is Nexerelin, but like Histidine said, that one should work as it extends CoreLifecyclePluginImpl and the only method in it is addScriptsIfNeeded()... Hmm.

Wait, that should work, right? No chance the game is replacing CoreLifecyclePluginImpl and then not calling the parent/super's onGameLoad because it's not explicitly included in the extension that replaced it?

Because if that's not it, I'm stumped. Maybe if something silently crashes beforeGameSave and onGameLoad?

----

Alternatively, reading through this .xml it looks like my hiddenMarkets list sometimes catches just a "ref", whatever that is, literally just a number -- and sometimes catches a "z" with all the details on a market plus, apparently, all of the campaign terrain, inside another market inside an "economy stepper".

Path looks like this (line broken for readability):
/CampaignEngine/scripts/data.scripts.campaign.bases.VayraRaiderBaseReaper/hiddenMarkets/Market[8]/economy/stepper/
econ/markets/Market/primaryEntity/orbit/f/where/o/saved/
LocationToken[147]/orbit/s/con/systems/Sstm[5]/o/saved/
Astrd[196]/cD/e/AsteroidBeltTerrainPlugin

Another, longer one, lives under a different market:
/CampaignEngine/scripts/data.scripts.campaign.bases.VayraRaiderBaseReaper/hiddenMarkets/Market[25]/economy/stepper/
econ/markets/Market/primaryEntity/orbit/f/where/o/saved/
LocationToken/orbit/s/con/systems/Sstm[4]/o/saved/
Plnt[2]/market/submarkets/Submarket[4]/f/production/gatheringPoint/admin/stats/fleet/cL/o/saved/
Flt[3]/ai/n/a/items/IDt[2]/i/e/dL/RouteManager/r/RouteData[33]/e/RtSeg[2]/t/cL/con/systems/Sstm[3]/o/saved/
Astrd[252]/cD/e/AsteroidBeltTerrainPlugin

So maybe my fixing that list will fix this? I'm gonna try it (actually already have), so fingers crossed?   

[Alex]

Did a quick search through all the mods posted in that modlist with Notepad++'s "find in files" and the only mod that has a coreLifecyclePlugin in its settings.json is Nexerelin, but like Histidine said, that one should work as it extends CoreLifecyclePluginImpl and the only method in it is addScriptsIfNeeded()... Hmm.

Wait, that should work, right? No chance the game is replacing CoreLifecyclePluginImpl and then not calling the parent/super's onGameLoad because it's not explicitly included in the extension that replaced it?

Because if that's not it, I'm stumped. Maybe if something silently crashes beforeGameSave and onGameLoad?

No, that should be fine. For any method not included in the extending class, the version in the parent class will be used - that's kind of a core thing about how inheritance works. And it's tough to see how beforeGameSave() could fail silently.

So, that's weird. A couple of things point towards CoreLifecyclePluginImpl.beforeGameSave() not being called for some reason - if it was just asteroids, that would be doubtful. But it's both asteroids and orbital_junk, the two things that method clears out.

Alternatively, reading through this .xml it looks like my hiddenMarkets list sometimes catches just a "ref", whatever that is, literally just a number -- and sometimes catches a "z" with all the details on a market plus, apparently, all of the campaign terrain, inside another market inside an "economy stepper".

Path looks like this (line broken for readability):
/CampaignEngine/scripts/data.scripts.campaign.bases.VayraRaiderBaseReaper/hiddenMarkets/Market[8]/economy/stepper/
econ/markets/Market/primaryEntity/orbit/f/where/o/saved/
LocationToken[147]/orbit/s/con/systems/Sstm[5]/o/saved/
Astrd[196]/cD/e/AsteroidBeltTerrainPlugin

That path basically doesn't matter and is not an indication of WHY something is in the savefile or what it relates to. It'll be super misleading if you treat it as such.

"ref" is a reference to another element that's found in full detail elsewhere in the savefile. For example, a market's data will be written out in full once, and then whenever something else refers to that market, it'll use "ref=<that market's savefile ID>"

"z" is the attribute which contains that unique (to the savefile, and only relevant to that specific savefile) id.

[Ishman]

Did a fast run with the only non utility mod being vayra's sector, completed one pirate bounty and dipped into four star systems total. Ctrl f the save and found 2000 instances of asteroids.

http://www.mediafire.com/file/7iihyaruabfad8p/campaign.xml/file

[Alex]

Thank you for giving this a try! Checked the file, it actually looks fine - the specific things that you'd want to look for is "Astrd" and "orbital_junk". Looks like neither is ending up in the save, at least at this point. The file size is also fine - around 10mb is about what I'd expect here.

(The occurrences of "asteroid" are legitimate, i.e. things like the terrain rather than individual asteroids; texture names, etc.)

[Ishman]

Gotcha, going to go through all the released mods to find the cause then - it should've already showed up since I did dip through a system with asteroid entities.

Edit: So far I believe I've ruled out BRDY, Interstellar Imperium, Arsenal Expansion, and Vayra's sector.

Edit2: Further info - the problem does not present itself quickly, it seems, I went through enabling one by one half of everything I've got - then enabled everything and did the same (Add fuels and supplies, then fly around between systems with asteroids, saving in hyperspace, in system, in hyperspace again, reloading every so often after a save). It doesn't seem to immediately present itself, so it might be something that you have to leave the game idling for a long time to start noticing, which will make it much harder to narrow down. It didn't seem to become apparent when spamming save/load/save copy.

[Alex]

This *seems* like something that would manifest immediately, and not even require flying around the various systems. Unless I'm wrong about the cause, of course, which is possible.

So the fact that you're been unable to reproduce it seems to suggest that the set of mods you're using doesn't cause the problem at all.

Just to confirm, you've tried every mod from this list?

Here's the list of mods I've been using:

Spoiler

{"enabledMods": [
  "$$$_trailermoments",
  "anotherportraitpack",
  "raccoonarms",
  "blackrock_driveyards",
  "CombatAnalytics",
  "chatter",
  "lw_radar",
  "istl_dam",
  "diableavionics",
  "DisassembleReassemble",
  "gs",
  "GMDA",
  "HMI",
  "HA",
  "junk_pirates_release",
  "lw_lazylib",
  "leadingPip",
  "ArkLeg",
  "luddenhance",
  "MagicLib",
  "Mayasuran Navy",
  "nexerelin",
  "ORA",
  "sun_ruthless_sector",
  "sad",
  "SCY",
  "shadow_ships",
  "swp",
  "bonomel_skilledup",
  "speedUp",
  "Sylphon_RnD",
  "tahlan",
  "underworld",
  "US",
  "URW",
  "vayrasector",
  "lw_version_checker",
  "audio_plus",
  "shaderLib"
]}

[close]

@erikem: From your experience, did the saves start out relatively large? E.G. say something larger than 20-30mb uncompressed?

Edit: If you happen to have a savefile where this is reproducible (both the campaign and the descriptor xml), I'd love to get my hands on it! And if you would also zip up and upload your mod folder somewhere, that would make my life a lot easier, though I know that's a bit to ask

[CynicJester]

I'm not sure if this is related or not, but I was having issues saving my game until I removed "luddenhance". I seem to recall what save files I got being bloated prior to removing it, but I already deleted them, so I can't say with complete certainty that it's related.

Edit: It seems unrelated, as upon checking my current save without the mod, there are some 163k instances of the word "Astrd", which seems excessive if they're supposed to be pruned.

Current modlist:

Spoiler

{"enabledMods": [
  "$$$_lightshow",
  "$$$_trailermoments",
  "lw_autosave",
  "blackrock_driveyards",
  "lw_radar",
  "lw_lazylib",
  "leadingPip",
  "MagicLib",
  "nexerelin",
  "ogaportraitpack",
  "shadow_ships",
  "swp",
  "bonomel_skilledup",
  "transponder-reminders",
  "URW",
  "vayrasector",
  "lw_version_checker",
  "audio_plus",
  "shaderLib",
  "Safety Override mod"
]}

[close]