Fractal Softworks Forum

Please login or register.

Login with username, password and session length
Advanced search  

News:

Starsector 0.97a is out! (02/02/24); New blog post: Simulator Enhancements (03/13/24)

Author Topic: [0.9.1a] Out of Memory trying to save - GC overhead limit exceeded  (Read 3139 times)

Nick XR

  • Admiral
  • *****
  • Posts: 712
    • View Profile

Running about 40 mods, the usual suspects of late game + Nexerlin + ~every faction I could get my hands on (No Combat Analytics though).  There are a fair number of colonies, most are old so size 7 or 8, and LOTS of fleets.

Steady-state SS uses about 5-6GB of memory, but on saving it runs out of memory with 38000 (~38GB) of RAM allocated to the JVM.  I have been slowly adding RAM to SS over the past week or so to keep it saving, but I've more or less hit my system limit, so now I'm here. 

Mod List:
Spoiler
Adjusted Sector [C:\Games\Starsector.9.1a-RC8\starsector-core\..\mods\Adjusted Sector v0.2.2]
ApproLight [C:\Games\Starsector.9.1a-RC8\starsector-core\..\mods\ApproLight]
raccoonarms [C:\Games\Starsector.9.1a-RC8\starsector-core\..\mods\Arsenal Expansion 1.5.3i]
audio_plus [C:\Games\Starsector.9.1a-RC8\starsector-core\..\mods\Audio Plus]
lw_autosave [C:\Games\Starsector.9.1a-RC8\starsector-core\..\mods\Autosave]
blackrock_driveyards [C:\Games\Starsector.9.1a-RC8\starsector-core\..\mods\Blackrock Drive Yards]
Celestial Mount Circle [C:\Games\Starsector.9.1a-RC8\starsector-core\..\mods\Celestial Mount Circle]
CombatAnalytics [C:\Games\Starsector.9.1a-RC8\starsector-core\..\mods\CombatAnalytics]
lw_console [C:\Games\Starsector.9.1a-RC8\starsector-core\..\mods\Console Commands]
COPS [C:\Games\Starsector.9.1a-RC8\starsector-core\..\mods\COPS]
istl_dam [C:\Games\Starsector.9.1a-RC8\starsector-core\..\mods\Dassault-Mikoyan Engineering 1.18]
diableavionics [C:\Games\Starsector.9.1a-RC8\starsector-core\..\mods\DIABLEAVIONICS]
edshipyard [C:\Games\Starsector.9.1a-RC8\starsector-core\..\mods\ED Shipyard]
shaderLib [C:\Games\Starsector.9.1a-RC8\starsector-core\..\mods\GraphicsLib]
HMI [C:\Games\Starsector.9.1a-RC8\starsector-core\..\mods\HMI]
deconomics [C:\Games\Starsector.9.1a-RC8\starsector-core\..\mods\Industrial.Evolution]
interestingportraitspack [C:\Games\Starsector.9.1a-RC8\starsector-core\..\mods\Interesting Portraits Pack]
Imperium [C:\Games\Starsector.9.1a-RC8\starsector-core\..\mods\Interstellar Imperium]
kadur_remnant [C:\Games\Starsector.9.1a-RC8\starsector-core\..\mods\Kadur Remnant]
lw_lazylib [C:\Games\Starsector.9.1a-RC8\starsector-core\..\mods\LazyLib]
ArkLeg [C:\Games\Starsector.9.1a-RC8\starsector-core\..\mods\Legacy of Arkgneisis]
MagicLib [C:\Games\Starsector.9.1a-RC8\starsector-core\..\mods\MagicLib]
Mayasuran Navy [C:\Games\Starsector.9.1a-RC8\starsector-core\..\mods\Mayasuran Navy]
Neutrino [C:\Games\Starsector.9.1a-RC8\starsector-core\..\mods\Neutrino corp]
nexerelin [C:\Games\Starsector.9.1a-RC8\starsector-core\..\mods\Nexerelin]
ORA [C:\Games\Starsector.9.1a-RC8\starsector-core\..\mods\ORA]
Polaris_Prime [C:\Games\Starsector.9.1a-RC8\starsector-core\..\mods\Polaris_Prime]
SCY [C:\Games\Starsector.9.1a-RC8\starsector-core\..\mods\SCY]
SEEKER [C:\Games\Starsector.9.1a-RC8\starsector-core\..\mods\SEEKER_UC]
swp [C:\Games\Starsector.9.1a-RC8\starsector-core\..\mods\Ship and Weapon Pack]
speedUp [C:\Games\Starsector.9.1a-RC8\starsector-core\..\mods\SpeedUp]
sun_starship_legends [C:\Games\Starsector.9.1a-RC8\starsector-core\..\mods\Starship Legends]
Sylphon_RnD [C:\Games\Starsector.9.1a-RC8\starsector-core\..\mods\sylphon]
tahlan [C:\Games\Starsector.9.1a-RC8\starsector-core\..\mods\tahlan]
THI [C:\Games\Starsector.9.1a-RC8\starsector-core\..\mods\Tiandong Heavy Industries 1.2.1a]
TORCHSHIPS [C:\Games\Starsector.9.1a-RC8\starsector-core\..\mods\Torchships]
underworld [C:\Games\Starsector.9.1a-RC8\starsector-core\..\mods\Underworld]
US [C:\Games\Starsector.9.1a-RC8\starsector-core\..\mods\UnknownSkies]
vayrasector [C:\Games\Starsector.9.1a-RC8\starsector-core\..\mods\Vayra's Sector]
vayrashippack [C:\Games\Starsector.9.1a-RC8\starsector-core\..\mods\Vayra's Ship Pack]
lw_version_checker [C:\Games\Starsector.9.1a-RC8\starsector-core\..\mods\Version Checker]
[close]

Command line params in case anyone cares:
Spoiler
java.exe -XX:CompilerThreadPriority=1 -XX:+CompilerThreadHintNoPreempt -XX:ParallelGCThreads=6 -XX:ConcGCThreads=3 -Djava.library.path=native\\windows  -Xms38000m -Xmx38000m -Xss4096k -server  UsualStuffAfterThis
[close]

Code
15053267 [Thread-19] INFO  com.fs.starfarer.campaign.save.CampaignGameManager  - Saving progress: 0.57523185
15053267 [Thread-19] INFO  com.fs.starfarer.campaign.save.CampaignGameManager  - Releasing context
15053268 [Thread-19] INFO  com.fs.starfarer.campaign.save.CampaignGameManager  - Released context
15053419 [Thread-4] INFO  com.fs.starfarer.campaign.save.CampaignGameManager  - Error saving game
15053420 [Thread-4] INFO  com.fs.starfarer.campaign.save.CampaignGameManager  - Renaming [campaign.xml.bak] to [campaign.xml]
15053420 [Thread-4] INFO  com.fs.starfarer.campaign.save.CampaignGameManager  - Renaming [descriptor.xml.bak] to [descriptor.xml]
15053420 [Thread-4] ERROR com.fs.starfarer.campaign.save.CampaignGameManager  - GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.lang.StringBuffer.toString(Unknown Source)
at com.thoughtworks.xstream.io.path.PathTracker.peekElement(PathTracker.java:133)
at com.thoughtworks.xstream.io.path.PathTracker.getPath(PathTracker.java:169)
at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:53)
at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)
at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.marshallField(AbstractReflectionConverter.java:274)
at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter$2.writeField(AbstractReflectionConverter.java:250)


Here's a snip of the memory usage over time while trying to save (when it fails)


Could some mods be doing some things that are pathologically bad for XStream to deal with (like a crazy object hierarchy)?  It seems odd it died 10GB short of the limit, perhaps it's trying to serialize some HUGE object.?

TJJ

  • Admiral
  • *****
  • Posts: 1905
    • View Profile
Re: [0.9.1a] Out of Memory trying to save - GC overhead limit exceeded
« Reply #1 on: June 27, 2020, 01:24:29 AM »

Is SS using xstream's default XML parser?
Might be worth trying others, as a properly functioning SAX-based parser shouldn't be creating such enormous peak memory usage.
Logged

martin

  • Ensign
  • *
  • Posts: 6
    • View Profile
Re: [0.9.1a] Out of Memory trying to save - GC overhead limit exceeded
« Reply #2 on: July 12, 2020, 11:15:37 PM »

-XX:+AggressiveHeap Does it for me. Still takes forever (10+ minutes) to save, but it does save and the saved game can be loaded. No idea why, Java is an evil sorcery as far as I am concerned. I am running with adjusted sector mod on a much weaker machine with only 8 gb of RAM, so it may perform better in your case.
Logged

Alex

  • Administrator
  • Admiral
  • *****
  • Posts: 23986
    • View Profile
Re: [0.9.1a] Out of Memory trying to save - GC overhead limit exceeded
« Reply #3 on: July 14, 2020, 08:26:31 AM »

Could some mods be doing some things that are pathologically bad for XStream to deal with (like a crazy object hierarchy)?  It seems odd it died 10GB short of the limit, perhaps it's trying to serialize some HUGE object.?

It seems probable/likely that there's a memory leak somewhere there, possibly in a custom "writeReplace()" method or some such. I suppose something "pathologically bad" is an outside possibility - something causing it to bug out - but that doesn't seem as likely. From prior experience, I'd expect that to take the form of "corrupted savefile", not *this*.

(It uses StAX for XML writing, btw.)

Does it generate a partial savefile? If so, the contents might be illuminating.
Logged

martin

  • Ensign
  • *
  • Posts: 6
    • View Profile
Re: [0.9.1a] Out of Memory trying to save - GC overhead limit exceeded
« Reply #4 on: July 14, 2020, 11:00:54 AM »

In my case it generates partial save file, sometimes ending in the middle of the line without printing the ">" symbol that I assume ends the line.
Logged

Alex

  • Administrator
  • Admiral
  • *****
  • Posts: 23986
    • View Profile
Re: [0.9.1a] Out of Memory trying to save - GC overhead limit exceeded
« Reply #5 on: July 14, 2020, 11:46:59 AM »

Gotcha - thank you for the info! It'll be interesting to see if it also produces a partial in Nick's case, which seems fairly different. In your case, it's plausible that it might run out of ram without there being a "problem" causing this. In Nick's, something has got to be going wrong to cause that high a usage of ram. So whether there's also a partial save there, and how big it is, might tell us something.
Logged

martin

  • Ensign
  • *
  • Posts: 6
    • View Profile
Re: [0.9.1a] Out of Memory trying to save - GC overhead limit exceeded
« Reply #6 on: July 15, 2020, 12:34:05 AM »

Why is it such a memory hog anyway?
Logged

Alex

  • Administrator
  • Admiral
  • *****
  • Posts: 23986
    • View Profile
Re: [0.9.1a] Out of Memory trying to save - GC overhead limit exceeded
« Reply #7 on: July 15, 2020, 09:47:59 AM »

Well, it's not one in vanilla. Mods (and players that run a whole bunch of mods at once) naturally push the envelope to just-next-to-the-breaking point. Which, I'm not saying that's bad! Just, kind of how it is. No matter what the baseline situation is, that seems like this would become the equilibrium point.

(Also, I strongly suspect there's a mod or two out there with some memory leaks - meaning 2x or higher memory use than needed - but that's not the easiest thing to pin down, either. Would basically require a player that's experiencing higher memory use to run jvisualvm, get a heap dump, and analyze it to see where the leak is coming from.

It's doable, just a fairly specialized skillset. Not the biggest of deals, though, since the game still runs, just needs a bunch of memory - due to how the game works, common memory leaks tend to be bounded. Nick's case, on the other hand...)
Logged

Nick XR

  • Admiral
  • *****
  • Posts: 712
    • View Profile
Re: [0.9.1a] Out of Memory trying to save - GC overhead limit exceeded
« Reply #8 on: July 15, 2020, 12:24:29 PM »

I got the itch to do some perf tuning so I was digging around in various tools to  try and see if I could solve my problem before posting here.  I've worked with several mod authors to fix issues of them holding onto CombatAPI related objects on the Campaign map.  FWIW Alex, most of the modding memory "leak" problems I found stemmed from a custom HullMod object holding on to one or more ShipAPIs which themselves hold onto AI objects which then hold onto the CombatEngine and then it's sad-panda time.  DarkRevanant has a pattern using 'getCustomData()' he uses to avoid this but if there were to be a CombatHullMod object that only lived in the combat life cycle it would be a bit harder for people to shoot their foot off.  But that's a lot of work to try and solve one problem vector for modders who can still easy wreck things innumerable other ways so...  The other memory leak I ran into was the more simple CombatAPI related stuff put into a static map and never cleared.  But even fixing the above issues didn't sort out my problem :-\

I think the problem has something to do with fleets (maybe something a mod is doing with fleets), maybe?  I say that because this is a late game problem when most of the colonies are size 8, so they generate LOTS of defensive fleets and then raiding fleets.


For what it's doing with the save file, it does write a partial file before it dies, the final line is:
Code
<items

Here are links to my last working save, and to the next failed in-progress save.
https://bitbucket.org/NickWWest/starsectorcombatanalytics/downloads/NickGoodSave.campaign.xml.zip
https://bitbucket.org/NickWWest/starsectorcombatanalytics/downloads/NickErrorOnSave.campaign.xml.inprogress.zip


I also have heap dumps if you're interested, but those are ~12GB...

Alex

  • Administrator
  • Admiral
  • *****
  • Posts: 23986
    • View Profile
Re: [0.9.1a] Out of Memory trying to save - GC overhead limit exceeded
« Reply #9 on: July 15, 2020, 01:10:20 PM »

Hmm. Just looking at the savefile, there are upwards of 2000 lines like this:

</data.scripts.campaign.bases.VayraRaiderNonEventFleetManager>

That, to me, seems indicative of some kind of major leak, or at least bug? I'm not familiar with the code base of the mod in question, but that many fleet managers seems like too many for a non-bugged out use case. I'm not sure if this is *the* problem; just something that stood out to me after randomly scrolling through your savefiles.

FWIW Alex, most of the modding memory "leak" problems I found stemmed from a custom HullMod object holding on to one or more ShipAPIs which themselves hold onto AI objects which then hold onto the CombatEngine and then it's sad-panda time.  DarkRevanant has a pattern using 'getCustomData()' he uses to avoid this but if there were to be a CombatHullMod object that only lived in the combat life cycle it would be a bit harder for people to shoot their foot off.  But that's a lot of work to try and solve one problem vector for modders who can still easy wreck things innumerable other ways so...  The other memory leak I ran into was the more simple CombatAPI related stuff put into a static map and never cleared.  But even fixing the above issues didn't sort out my problem :-\

Ah, interesting - thank you for digging into it! Do you mean CampaignEngine, though? Though I suppose holding on to a CombatEngine instance would be a problem, too, just seems like it'd be a much, much smaller one. And, yeah, using getCustomData() is the right pattern for this; that's what vanilla does as well.

I guess that hullmods are a common leak vector for mods, yeah... from what I've seen, it's also anything to do with static variables in any sort of script, combat or campaign.
Logged

Alex

  • Administrator
  • Admiral
  • *****
  • Posts: 23986
    • View Profile
Re: [0.9.1a] Out of Memory trying to save - GC overhead limit exceeded
« Reply #10 on: July 15, 2020, 03:05:47 PM »

Motivated by this thread, as well as the conversation via PM:

https://twitter.com/amosolov/status/1283522450245013506
Logged

Rotok

  • Ensign
  • *
  • Posts: 13
    • View Profile
Re: [0.9.1a] Out of Memory trying to save - GC overhead limit exceeded
« Reply #11 on: December 08, 2022, 11:27:31 PM »

This added to the vmparams file combined with using clearcommands (csmartclear) for Console Commands mod allowed me to save my game (thought at this point, I should just restart with a smaller sector and much less factions running around lol).

-XX:+AggressiveHeap Does it for me. Still takes forever (10+ minutes) to save, but it does save and the saved game can be loaded. No idea why, Java is an evil sorcery as far as I am concerned. I am running with adjusted sector mod on a much weaker machine with only 8 gb of RAM, so it may perform better in your case.
Logged