Fractal Softworks Forum

Please login or register.

Login with username, password and session length
Pages: [1] 2

Author Topic: GC Overhead Limit Exceeded.  (Read 13984 times)

Blaze

  • Commander
  • ***
  • Posts: 219
    • View Profile
GC Overhead Limit Exceeded.
« on: March 26, 2016, 10:36:37 AM »

So this is obviously an Out of Memory error, but I've set 4GB of RAM to it on a 16GB computer; so I think it might be a memory leak.

It seems to happen only during saving, and only after a certain amount of time passes.
Spoiler
2260543 [Thread-4] ERROR com.fs.starfarer.combat.CombatMain  - java.lang.OutOfMemoryError: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
   at java.util.Arrays.copyOfRange(Unknown Source)
   at java.lang.StringBuffer.toString(Unknown Source)
   at com.thoughtworks.xstream.io.path.PathTracker.peekElement(PathTracker.java:133)
   at com.thoughtworks.xstream.io.path.PathTracker.getPath(PathTracker.java:169)
   at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:53)
   at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
   at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)
   at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.mars hallField(AbstractReflectionConverter.java:256)
   at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter$2.writeField(AbstractReflectionConverter.java:232)
   at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter$2.<init>(AbstractReflectionConverter.java:195)
   at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.doMa rshal(AbstractReflectionConverter.java:141)
   at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.mars hal(AbstractReflectionConverter.java:89)
   at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
   at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
   at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)
   at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:88)
   at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.wri teItem(AbstractCollectionConverter.java:64)
   at com.thoughtworks.xstream.converters.collections.CollectionConverter.marshal(CollectionConverter.java:74)
   at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
   at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
   at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)
   at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.mars hallField(AbstractReflectionConverter.java:256)
   at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter$2.writeField(AbstractReflectionConverter.java:232)
   at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter$2.<init>(AbstractReflectionConverter.java:195)
   at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.doMa rshal(AbstractReflectionConverter.java:141)
   at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.mars hal(AbstractReflectionConverter.java:89)
   at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
   at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
   at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)
   at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.mars hallField(AbstractReflectionConverter.java:256)
   at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter$2.writeField(AbstractReflectionConverter.java:232)
   at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter$2.<init>(AbstractReflectionConverter.java:195)
[close]

Mods are:
AI War
Autonomous Ships
Blackrock Driveyards
Combat Chatter
Console Commands
Diable Avionics
Interstellar Imperium
Lazylib
Nexerelin
Shaderlib
Junk Pirates/ASP/PACK
Neutrino Corp
Metelson Industries
Portrait Pack
Save transfer
Console commands
SCY Nation
Shadowyards Reconstruction Authority
Knights Templar
Tiandong Heavy Industries

Game version 7.2a (Although I did have this issue in 7.1 as well; I chalked it up to having a craptop).
All mods were updated on March 23.
Logged

Alex

  • Administrator
  • Admiral
  • *****
  • Posts: 24114
    • View Profile
Re: GC Overhead Limit Exceeded.
« Reply #1 on: March 26, 2016, 10:51:47 AM »

It might indeed be a leak. Unfortunately, this is a tough one to figure out. It could be a vanilla memory leak, though I do keep an eye on it and run tests to try and spot any that slip by.

It could also be from a mod - unfortunately, it's *very* easy to code something in a mod, without a second thought, that would leak an entire campaign's worth of data every time you save or load.

That it happens during saving isn't necessarily an indicator of a problem with saving - saving just takes extra memory, and if an ongoing leak has increased the memory use to a point where there is enough to keep working, but not enough to save, then you'd see this issue.

... and only after a certain amount of time passes.

Is just that enough, or is save/loading prior to this involved as well? That'd be an interesting data point.


Again, though, this is tough to track down. Just as far as probability, I'd say it's more likely that there's a leak in a mod somewhere, but I certainly can't rule out a vanilla leak, either. I strongly suspect there is a leak somewhere, though, because 4GB seems excessive and always has to me.

I wrote up the instructions to use jvisualvm to find these kinds of things a while back:
http://fractalsoftworks.com/forum/index.php?topic=7690.msg128363#msg128363

But you'd have to be pretty comfy with dev tools and/or java to do that, I think. That's more aimed at modders - I think it might be worthwhile for people to check for leaks in this way, since, as I mentioned earlier, it can be very easy to get a save/load leak from code that otherwise works perfectly.

Logged

Blaze

  • Commander
  • ***
  • Posts: 219
    • View Profile
Re: GC Overhead Limit Exceeded.
« Reply #2 on: March 26, 2016, 01:56:29 PM »

It only seems to happen while saving, it gets stuck on the screen for about a minute and then crashes.

If I load and immediately save, it doesn't seem to happen. Though it could just be that enough time hasn't passed.
Logged

Dark.Revenant

  • Admiral
  • *****
  • Posts: 2806
    • View Profile
    • Sc2Mafia
Re: GC Overhead Limit Exceeded.
« Reply #3 on: March 26, 2016, 04:39:44 PM »

Without access to the source, I can't say for sure, but I believe this is a vanilla issue.

Alex, check the Jump Point implementation and the mechanic where entities are enumerated when you mouse over the jump point.  A data structure in there seems to be holding onto enough objects to basically keep the previous save's entire universe in memory (star systems, hyperspace, and everything contained therein).  CampaignEngine is not duplicated.

Of course, there could be additional problems on top of this, but this one is readily visible on a fresh save.
Logged

Alex

  • Administrator
  • Admiral
  • *****
  • Posts: 24114
    • View Profile
Re: GC Overhead Limit Exceeded.
« Reply #4 on: March 26, 2016, 04:54:38 PM »

You mean that the data in the save xml file tends to, in large portion, be within a jump-point element? That's normal; it's just pulling out the object graph and a jump point happens to be one of the earlier entities, so a lot of stuff goes under if even if conceptually that's not how you might structure it. But an object graph is just that, a graph, not a tree, so odd-looking things can be the "top" if it's automatically pulled out and serialized.

Or are you actually spotting data from another campaign instance inside a save file?


Honestly, I strongly suspect a mod here, *or* an issue I haven't run into during normal development. Generally, when a campaign instance leaks, you'll start to notice tiny stutters when save/loading and occasionally during gameplay. Once you get a feel for what those are like, they don't slip by too easily, which is why I feel fairly confident in making that statement.

That's not to say that vanilla has no leaks under any circumstances. It's possible/probably that it does. However, out of memory crash reports are frequent enough that it seems like - if they are indeed caused by leaks - then the conditions for the leaks are pretty common, making it less likely (but, of course, not impossible) that I haven't run into it during dev.


(Being paranoid, just did some testing w/ save-loading while showing the jump point tooltip and looked through it w/ jvisualvm; nothing untoward. The campaign-object tooltip manager is transient and doesn't get saved/isn't tied to the UI that persists across saves, but doesn't hurt to double-check.)
Logged

Dark.Revenant

  • Admiral
  • *****
  • Posts: 2806
    • View Profile
    • Sc2Mafia
Re: GC Overhead Limit Exceeded.
« Reply #5 on: March 26, 2016, 05:11:03 PM »

If that's the case then SS+ is most likely at fault.  It could be literally any campaign class in there.

This is a huge problem because I must fix this.  I refuse to make another release if this is happening because of how many issues this causes.  I feel like I'm slowing the development of the game due to all of the RAM reports you get.
Logged

Alex

  • Administrator
  • Admiral
  • *****
  • Posts: 24114
    • View Profile
Re: GC Overhead Limit Exceeded.
« Reply #6 on: March 26, 2016, 05:18:25 PM »

If that's the case then SS+ is most likely at fault.  It could be literally any campaign class in there.

I'm not 100% sure, right? Could still be something in vanilla. But I'd really appreciate you taking a look, using jvisualvm + the instructions from that link a couple of posts ago. It actually doesn't take very long once you've got the steps down; end up doing this myself fairly often if I suspect anything at all fishy :)

(I sure hope that jvisualvm works on obfuscated code - double-checked, yes it does.)


I feel like I'm slowing the development of the game due to all of the RAM reports you get.

No worries on that front - if I see an out-of-memory for a heavy modded game and there's no new information compared to previous reports, I don't really spend very much time looking at it, since there's not much I *can* look at, in that case.
Logged

Weltall

  • Admiral
  • *****
  • Posts: 774
    • View Profile
Re: GC Overhead Limit Exceeded.
« Reply #7 on: March 26, 2016, 05:50:10 PM »

I had that on 0.7.1a myself and i was monitoring it. Every save would raise the memory usage by 1GB every time and eventually reach 6+ GB which meant taking over all my ram. If I did not save, i could play for hours with no problem. I had Nexerelin on the maximum size of course when it happened that. With 0.7.2a thankfully that is not happening. I have been playing for long on a game and ram gets stuck at 5+GB, but never goes to 6. If anything through, I switched from Java 8u73 to Java 7u97. I could not try Java 7 on 0.7.1a, because probably one of my mods wanted 8? If anything the game would crash and I never checked which one it was.
Logged
Ignorance is bliss..

Blaze

  • Commander
  • ***
  • Posts: 219
    • View Profile
Re: GC Overhead Limit Exceeded.
« Reply #8 on: March 26, 2016, 06:25:46 PM »

If that's the case then SS+ is most likely at fault.  It could be literally any campaign class in there.
In the example above I'm not using SS+. It might be a contributor, but it's not the cause.
Logged

Dark.Revenant

  • Admiral
  • *****
  • Posts: 2806
    • View Profile
    • Sc2Mafia
Re: GC Overhead Limit Exceeded.
« Reply #9 on: March 26, 2016, 07:40:55 PM »

I'm seeing the problem on vanilla, Alex.  The steps to reproduce:

1. Start new save, take heap snapshot.
2. Enter battle, exit battle.
3. Load save, take heap snapshot.
4. Observe the duplicate Hyperspaces, Star Systems, etc.

As such, it is impossible for me to differentiate between vanilla and modded problems, so I'll stop my investigation.
Logged

Alex

  • Administrator
  • Admiral
  • *****
  • Posts: 24114
    • View Profile
Re: GC Overhead Limit Exceeded.
« Reply #10 on: March 26, 2016, 07:49:58 PM »

Hmm. Would you mind elaborating on the "observe the duplicate..." step? I'm not seeing any duplicates of the above trying this just now. Each heap dump has precisely 1 CampaignEngine and 1 Hyperspace instance, for example.


Edit: tried this on a proper Windows install of the game rather than the dev version (paranioa), but still not seeing anything wrong. Just 1 instance of each of those classes, etc.

You're not comparing whether the CampaignEngine from heap dump 1 is the same object as the CampaignEngine from heap dump 2, are you? They're not the same and they're not supposed to be, loading creates a new CampaignEngine object and the old one is freed up.

Edit #2: Side note - if you happen to be stepping through something with a debugger, it may hold on to references to the code involved after you continue executing, which'll look like memory leaks initially but, well, is just the debugger. Also, if you do the "find GC root" step from the steps I wrote up, you'll be able to see exactly what's preventing any given instance of CampaignEngine etc from being garbage collected.
« Last Edit: March 26, 2016, 08:16:22 PM by Alex »
Logged

Dark.Revenant

  • Admiral
  • *****
  • Posts: 2806
    • View Profile
    • Sc2Mafia
Re: GC Overhead Limit Exceeded.
« Reply #11 on: March 26, 2016, 08:19:20 PM »

I'm observing multiple instances of Hyperspace, StarSystem, etc.  CampaignEngine still has 1 instance.  This is after a GC round, too.  Heap 2 has about 75% more instances of objects overall, and is also about 75% larger in bytes.  This effect does not compound; the number of instances does not continue to rise with additional loads.

Worth mentioning is that I'm debugging with JDK 8 and using JRE 8 x64 for the runtime environment.  I'm not debugging it.

In what seems to be all cases, the thing preventing it from being garbage collected is a long tree going to whatever the "if" class is.

Economy leak:


Hyperspace leak:
« Last Edit: March 26, 2016, 08:28:49 PM by Dark.Revenant »
Logged

Alex

  • Administrator
  • Admiral
  • *****
  • Posts: 24114
    • View Profile
Re: GC Overhead Limit Exceeded.
« Reply #12 on: March 26, 2016, 08:37:22 PM »

Hmm - both of these look like leaks from some sort of Java tooling, since both are ultimately being prevented from being GC'ed by a reference from what looks like a different thread. Might be a Java 8 thing, might be related to the IDE you're launching it from (if any), but I don't think this is a problem in any case. Hard to be sure without digging around a bit, but at least that's what it looks like to me from the screenshots.
« Last Edit: March 26, 2016, 08:39:06 PM by Alex »
Logged

Dark.Revenant

  • Admiral
  • *****
  • Posts: 2806
    • View Profile
    • Sc2Mafia
Re: GC Overhead Limit Exceeded.
« Reply #13 on: March 26, 2016, 08:46:33 PM »

Might be the concurrent GC I set in the params.  In theory, that GC should eventually free up the blocked stuff.  If it doesn't, that indicates a bug in the GC implementation, or weird architecture in Starsector.

And now Starsector isn't showing up in jvisualvm.
Logged

Alex

  • Administrator
  • Admiral
  • *****
  • Posts: 24114
    • View Profile
Re: GC Overhead Limit Exceeded.
« Reply #14 on: March 26, 2016, 08:53:02 PM »

Might be the concurrent GC I set in the params.  In theory, that GC should eventually free up the blocked stuff.

Right - another odd thing here is that if it was an actual normal hyperspace instance, I'd expect to see a 2nd CampaignEngine instance, as well. So we can probably chalk it up to the vagaries of the GC.


And now Starsector isn't showing up in jvisualvm.

Weird, haven't had that happen myself. Reboot, perhaps?
Logged
Pages: [1] 2