Is there anything I can do? Is there anything responsible I could monitor?I've got this exact same issue: ton of mods, larger fleets, lots of RAM allocated to the game, but then serious drops in FPS after battles until getting into the next battle, where the game will run fine, then get even laggier after getting back to the stratmap. I tried to figure out what it was by a little testing here and there, and at first I thought it was just a problem when two big fleets duked it out with one another, but after a while I noticed that it seemed to have something to do with the afterbattle looting as it seemed to get worse when I had more varieties of item in the cargohold, but don't quote me on that.
If you can share your mod list, I think that'd be helpful - this indeed sounds like some kind of leak or similar issue. Knowing which mods are involved could help narrow it down over time.Every mod that works with the current game version and doesn't overlap. I am currently running about 90 mods.
Depending on how comfortable you are with that sort of stuff, there are some instructions here:Comfortable enough! Will do that extensively tomorrow and report back.
http://fractalsoftworks.com/forum/index.php?topic=7690.msg128363#msg128363
Basically, best-case scenario is this would zero in on a specific Java class in a specific mod.
I noticed that it seemed to have something to do with the afterbattle looting as it seemed to get worse when I had more varieties of item in the cargohold, but don't quote me on that.Quoted you on that on principle, and, that is a valid pointer. I will look out for that.
You can also use the ConsoleCommands to take a heap dump, if you do that and put the dump in a public place I can take a look at it too. I'm having similar issues (https://fractalsoftworks.com/forum/index.php?topic=18753.msg292800#msg292800) so another data point is interesting to me.
(also once you have a dump you can use EclipseMAT too, or Visual VM)
PS: You might wanna update that post with the links to VisualVM, the usage is analog enough that I could replicate all steps you mentioned in that program.
PS: Never had a GC issue, only degrading performance. Was always able to save, even in 3FPS scenarios.
Interesting. It taking that long - you mentioned five minutes! - still seems to indicate that there's something going on to slow it down, though. It's possible that the leak just isn't quite big enough to make it crash while saving but just big enough to make it slow, but, yeah, that'd be a fine line for it to walk. The really low campaign framerate seems to be almost be more indicative of some kind of cumulative problem with, say, the same script getting repeatedly added to the game or something that just over time adds up and massively slows it down.My lategame saves are well above 70MB, with my current one being roughly 82MB and climbing. I haven't had the time to start a new game yet to hunt for "that" behavior again, but there's a weekend approaching.
Semi-related note: how big are your savefiles? The size of the campaign.xml, specifically.
You could still try that VisualVM monitoring that Alex mentioned, see if you have a leak or nah. Considering that Starsector was first released (in Beta) in 2011, it is entirely possible that our contemporary rigs can keep going with a leaked campaign engine. I could imagine that current gen rigs simply shoulder that and keep going, but Alex might be able to shed light on that. I am just making *** up at this point.
I can't find that on the log but I typed them out:SpoilerAI Core Production 1.0.0
Audio Plus 1.1.1x (old version which I modified to include other music)
Better Colonies 1.4
Commissioned Crews 1.9
Flux Reticle 1.0.1
GraphicsLib 1.4.2
HullMods Expansion 4.2a (should be the latest version before the author asked it to be removed from the forums)
Hyperdrive 2.0.0
Industrial.Evolution 1.7.d
Interstellar Imperium 2.2.1
Karlsson Heirloom 0.12
Lazylib 2.4f
Leading Pip 1.9.0
Logistics Notifications 1.3
Luddic Enhancement 1.2.2
MagicLib 0.29
Nexerelin 0.9.6e
Personal Adjustments (My own private mod which is basically just settings changes)
Player Gate Construction 1.1.1
Polaris Prime 0.23 RC-1
Secrets of the Frontier Prerelease 7
Shadowy Broker 0.1.5
Ship/Weapons Pack 1.11.0
SkilledUp 1.1
SpeedUp 0.6.1
Tahlan Shipworks 0.3.13
The Knights Templar 0.9.9b
Tiandong Heavy Industries 1.2.1a
Transfer All Items 1.1
Underworld 1.4.4
Vayra's Sector 3.1.5
Vesperion Combine 1.2.0
Weapons Group Controls 1.1.0[close]
...
Then those two lists together equals this one, where only the mods that are in all three lists are:
...
I'm curious in knowing if anyone that has been getting these slowdowns can report through the task manager if java is using more than 3G. I have a feeling that the game wants to use more memory but it can't for whatever reason, even when set to.I've had Starsector at any number from 4GB to 7.8GB (when I had given it 8GB) of RAM. Sorry.
I just have no indication on what I could investigate further to confirm/rule out that it's Vayra's.
Any ideas on what to look for/how to look for the source of this?
More pointers towards Vayra's Sector? I just have no indication on what I could investigate further to confirm/rule out that it's Vayra's.
Any ideas on what to look for/how to look for the source of this?
The mod in question would be Vayra's Sector, which spawns these IBB bounty missions.
Edit: 15 minutes later, letting it sit for a while, GC seems to have cleaned up some, but not all of them. Performance has recovered from 1 FPS to ~8FPS. At this time Starsector us using 5.6GB out of 7GB assigned.
"Thread-4" prio=6 tid=0x0000000019fa8800 nid=0x59cc runnable [0x000000001a87e000]
java.lang.Thread.State: RUNNABLE
at com.fs.starfarer.settings.StarfarerSettings$1.getCommoditySpec(Unknown Source)
at com.fs.starfarer.campaign.Faction.isIllegal(Unknown Source)
at com.fs.starfarer.campaign.econ.Market.isIllegal(Unknown Source)
at exerelin.campaign.intel.missions.Nex_ProcurementMissionIntel.pickMarket(Nex_ProcurementMissionIntel.java:82)
at com.fs.starfarer.api.impl.campaign.intel.ProcurementMissionIntel.<init>(ProcurementMissionIntel.java:81)
at exerelin.campaign.intel.missions.Nex_ProcurementMissionIntel.<init>(Nex_ProcurementMissionIntel.java:24)
at exerelin.campaign.intel.missions.Nex_ProcurementMissionCreator.createMissionIntel(Nex_ProcurementMissionCreator.java:10)
at com.fs.starfarer.api.impl.campaign.intel.GenericMissionManager.createEvent(GenericMissionManager.java:132)
at com.fs.starfarer.api.impl.campaign.intel.BaseEventManager.advance(BaseEventManager.java:141)
at com.fs.starfarer.campaign.CampaignEngine.advance(Unknown Source)
at com.fs.starfarer.campaign.CampaignState.advance(Unknown Source)
at com.fs.starfarer.BaseGameState.traverse(Unknown Source)
at com.fs.state.AppDriver.begin(Unknown Source)
at com.fs.starfarer.combat.CombatMain.main(Unknown Source)
at com.fs.starfarer.StarfarerLauncher$1.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
The first way to do this is to right-click on the appropriate Java process and select the option "Thread Dump." This will generate a thread dump file whose name appears under the selected Java process as shown in the following screen snapshot.
Another thing you could try is start the CPU sampler in visualvm. I'm not sure exactly what the steps would be for visual vm specifically, but for jvisualvm there's a "sampler" tab where you can press a "CPU" button to begin sampling. It's a more efficient (but probably a bit harder to interpret?) way of doing the same thing, basically. It'll show which methods are taking up the most CPU time by doing sampling (rather than full-on profiling, which is more exact in some circumstances but also bogs the application down by a factor of 10 or more).
SpoilerStill not definitive, but here's some more data:
Baseline, flying around with no slowdown, I see:
54% of time is in com.fs.starfarer.campaign.CampaignEngine.advance()
35% of time is in org.lwjgl.opengl.Display.update()
7.5% of time is in com.fs.starfarer.campaign.CampaignState.render()
With slowdown, the breakdown is instead:
54% CampaignEngine.advance()
25% Display.update()
14% CampaignState.render()
Hm. Looks like there's something suspicious under CampaignState.render(). Let's break that down another level.
Baseline, no slowdown:
com.fs.starfarer.campaign.CampaignEngine.render(): 3.2%
com.fs.starfarer.ui.interface.render: 3%
and a few other smaller things make it all round up to 7.5% total for CampaignState.render.
With slowdown:
com.fs.starfarer.ui.interface.render: 8.6%
...Okay, so it looks like whatever the problem is, it's something to do with UI rendering? (Trying to track the stack traces down further did not, unfortunately, yield any further information or any sort of obvious smoking gun.)
This was, again, a slowdown triggered after end of battle. Though, with your comment on invisible tooltips hanging around, I'm wondering if there's a way for such a thing to be generated by an end-of-battle collection screen or something? Looks like I may need to do memory sampling in addition to CPU sampling... but not tonight. This morning? Whichever; I need to get some sleep.[close]
I did reduce my battle size from 500 to 400 intermittently, though. Maybe that helps? Maybe that averts it from reoccurring? Show of hands, what's your battle sizes?2000. I did a little tweak because I wanted bigger battles with more stuff going on, but I get the problem even with small battles of 500-points and just had it again in a whole new save with completely updated mods and loads of other stuff going on. Ended with a fleet wipe, but when I respawned, pure stutter.
Profiling continues to turn up nothing of interest, and console still indicates memory use is a fraction of the available amount.Did you look at the distribution of free space inside heap space? We might be fine in eden and survivor space but running out of tenured space (or worse - permanent space).
...
I have the inkling that the slowdown stems from some unknown issue in the JVM.
(IIRC one of - not the only, there was some weird hard crashing, too - the issues with 8 is that its GC settings are by default different and can cause actual frame drops especially with larger allocations. There's, again iirc, some set of JVM parameters that could be added to use the default Java 7 GC behavior.)Java7 uses serial collection, while Java8 uses parallel (serial if you use 32bit or single CPU machine). Neither seems good for heavily modded game with large heap sizes. Perhaps we should try running with G1 instead, as that's optimal for large memory (5-6G or larger) and low pause time (0.5s or less)?
...
Back on Java 7 (for testing purposes): I was able to reproduce in my game without Nexerelin or Tahlan, albeit with a significant number of other mods and I had to spawn + nuke a bunch of fleets with console to get a really noticeable effect.
The game uses parallelGC by default, but the algorithm does not even matter here. I would expect the users to get an OOM error instead of gradually degrading FPS if the issue is memory related.
-Xloggc:../../../logs/gc.log -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
I only tried it once, and don't really recall the sequence of events. There was no attempt at creating a reproducible test case, although I could try doing so using the save.Back on Java 7 (for testing purposes): I was able to reproduce in my game without Nexerelin or Tahlan, albeit with a significant number of other mods and I had to spawn + nuke a bunch of fleets with console to get a really noticeable effect.
Just to clarify,
You manually spawned one fleet using console, then fought the fleet. Then went to a market to repair own fleet? Then performed a number repetitions of the cycle? How many repetitions? Anything else? What was the approximate size of both fleets?
Would you consider that to be a proper reproducible test case at this point?
It's not the rendering that is costly, it's calculating all the object interactions.
Ah, my main point is that this looks like a completely separate issue from what's discussed in the thread.
I'm curious in knowing if anyone that has been getting these slowdowns can report through the task manager if java is using more than 3G. I have a feeling that the game wants to use more memory but it can't for whatever reason, even when set to.
No solution for the post combat problem. Vayras' sector has been identified as culprit for most campaign layer lag issues due to fleet spam.
This can be alleviated by using ClearCommands, https://fractalsoftworks.com/forum/index.php?topic=19210, cSmartClear, but it's nothing other than a band aid.
For new saves, do this before starting:
Go to Vayra's Sector\data\config\vayraRaiders
Open warhawk_republic.json
find the line:
"spawnNonEventFleets":true, # OPTIONAL Boolean, default false
# Causes small, NON-EVENT-RELATED fleets to spawn from ALL markets owned by the faction (i.e. not just bases created by this framework)
and set it to false
do the same for hegemony.json
Is this issue still relevant in 0.95a?I'm still getting it on 0.95a (and IIRC it happened on my unmodded/minimally modded run). Haven't tested if the Java 8 workaround still works.
Is this issue still relevant in 0.95a?I'm still getting it on 0.95a (and IIRC it happened on my unmodded/minimally modded run). Haven't tested if the Java 8 workaround still works.
java.exe -XX:+ShowMessageBoxOnError -XX:+UnlockExperimentalVMOptions -XX:+DisableExplicitGC -XX:+AggressiveOpts -XX:+TieredCompilation -XX:+UseG1GC -XX:InitialHeapSize=2048m -XX:MaxMetaspaceSize=2048m -XX:MaxNewSize=2048m -XX:+ParallelRefProcEnabled -XX:G1NewSizePercent=5 -XX:G1MaxNewSizePercent=10 -XX:G1ReservePercent=5 -XX:InitiatingHeapOccupancyPercent=80 -XX:G1HeapWastePercent=5 -XX:MaxGCPauseMillis=100 -XX:G1HeapRegionSize=2M -XX:+UseStringDeduplication -Xms10512m -Xmx10512m -Xss2m
with these options starsector wont even start. What's you vmparams like?
./jre_linux/bin/java -server -XX:+UseLargePages -XX:+UseParallelGC -XX:-UseGCOverheadLimit -XX:CompilerThreadPriority=1 -XX:+CompilerThreadHintNoPreempt -Djava.library.path=./native/linux -Xmn1g -Xms4g -Xmx4g -Xss4m -classpath janino.jar:commons-compiler.jar:commons-compiler-jdk.jar:starfarer.api.jar:starfarer_obf.jar:jogg-0.0.7.jar:jorbis-0.0.15.jar:json.jar:lwjgl.jar:jinput.jar:log4j-1.2.9.jar:lwjgl_util.jar:fs.sound_obf.jar:fs.common_obf.jar:xstream-1.4.10.jar -Dcom.fs.starfarer.settings.paths.saves=./saves -Dcom.fs.starfarer.settings.paths.screenshots=./screenshots -Dcom.fs.starfarer.settings.paths.mods=./mods -Dcom.fs.starfarer.settings.paths.logs=. -Dcom.fs.starfarer.settings.linux=true com.fs.starfarer.StarfarerLauncher
#./jre_linux/bin/java -server -XX:-UseGCOverheadLimit -XX:+UseParallelGC -XX:CompilerThreadPriority=1 -XX:+CompilerThreadHintNoPreempt -Djava.library.path=./native/linux -Xms4g -Xmx4g -Xss4m -Xmn64m -classpath janino.jar:commons-compiler.jar:commons-compiler-jdk.jar:starfarer.api.jar:starfarer_obf.jar:jogg-0.0.7.jar:jorbis-0.0.15.jar:json.jar:lwjgl.jar:jinput.jar:log4j-1.2.9.jar:lwjgl_util.jar:fs.sound_obf.jar:fs.common_obf.jar:xstream-1.4.10.jar -Dcom.fs.starfarer.settings.paths.saves=./saves -Dcom.fs.starfarer.settings.paths.screenshots=./screenshots -Dcom.fs.starfarer.settings.paths.mods=./mods -Dcom.fs.starfarer.settings.paths.logs=. -Dcom.fs.starfarer.settings.linux=true com.fs.starfarer.StarfarerLauncher
systemd-run --scope -p MemoryMax=7G -p MemorySwapMax=32M --user ./starsector.sh &>/dev/null &
while [ TRUE ]; do
echo 'Waiting 1 minute...'
sleep 55s
echo 'Clearing cache...'
sync && echo 3 | sudo tee /proc/sys/vm/drop_caches > /dev/null
sleep 5s
free -h
done
00:00.0 Host bridge: Intel Corporation Broadwell-U Host Bridge -OPI (rev 09)
Subsystem: Matsushita Electric Industrial Co., Ltd. Device 8338
Kernel driver in use: bdw_uncore
00:02.0 VGA compatible controller: Intel Corporation HD Graphics 5500 (rev 09)
Subsystem: Matsushita Electric Industrial Co., Ltd. Device 8338
Kernel driver in use: i915
Kernel modules: i915
00:03.0 Audio device: Intel Corporation Broadwell-U Audio Controller (rev 09)
Subsystem: Matsushita Electric Industrial Co., Ltd. Device 8338
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
00:04.0 Signal processing controller: Intel Corporation Broadwell-U Processor Thermal Subsystem (rev 09)
Subsystem: Matsushita Electric Industrial Co., Ltd. Device 8338
Kernel modules: processor_thermal_device_pci_legacy
00:14.0 USB controller: Intel Corporation Wildcat Point-LP USB xHCI Controller (rev 03)
Subsystem: Matsushita Electric Industrial Co., Ltd. Device 8338
Kernel driver in use: xhci_hcd
Kernel modules: xhci_pci
00:16.0 Communication controller: Intel Corporation Wildcat Point-LP MEI Controller #1 (rev 03)
Subsystem: Matsushita Electric Industrial Co., Ltd. Device 8338
Kernel driver in use: mei_me
Kernel modules: mei_me
00:1b.0 Audio device: Intel Corporation Wildcat Point-LP High Definition Audio Controller (rev 03)
Subsystem: Matsushita Electric Industrial Co., Ltd. Device 8338
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
00:1c.0 PCI bridge: Intel Corporation Wildcat Point-LP PCI Express Root Port #1 (rev e3)
Subsystem: Matsushita Electric Industrial Co., Ltd. Device 8338
Kernel driver in use: pcieport
00:1c.2 PCI bridge: Intel Corporation Wildcat Point-LP PCI Express Root Port #3 (rev e3)
Subsystem: Matsushita Electric Industrial Co., Ltd. Device 8338
Kernel driver in use: pcieport
00:1d.0 USB controller: Intel Corporation Wildcat Point-LP USB EHCI Controller (rev 03)
Subsystem: Matsushita Electric Industrial Co., Ltd. Device 8338
Kernel driver in use: ehci-pci
00:1f.0 ISA bridge: Intel Corporation Wildcat Point-LP LPC Controller (rev 03)
Subsystem: Matsushita Electric Industrial Co., Ltd. Device 8338
Kernel driver in use: lpc_ich
Kernel modules: lpc_ich
00:1f.2 SATA controller: Intel Corporation Wildcat Point-LP SATA Controller [AHCI Mode] (rev 03)
Subsystem: Matsushita Electric Industrial Co., Ltd. Device 8338
Kernel driver in use: ahci
00:1f.3 SMBus: Intel Corporation Wildcat Point-LP SMBus Controller (rev 03)
Subsystem: Matsushita Electric Industrial Co., Ltd. Device 8338
Kernel driver in use: i801_smbus
Kernel modules: i2c_i801
00:1f.6 Signal processing controller: Intel Corporation Wildcat Point-LP Thermal Management Controller (rev 03)
Subsystem: Matsushita Electric Industrial Co., Ltd. Device 8338
Kernel driver in use: intel_pch_thermal
Kernel modules: intel_pch_thermal
02:00.0 Network controller: Intel Corporation Wireless 7265 (rev 59)
Subsystem: Intel Corporation Dual Band Wireless-AC 7265
Kernel driver in use: iwlwifi
Kernel modules: iwlwifi
total used free shared buff/cache available
Mem: 7.7Gi 1.1Gi 5.9Gi 232Mi 618Mi 6.0Gi
Swap: 0B 0B 0B
lsmem:
RANGE SIZE STATE REMOVABLE BLOCK
0x0000000000000000-0x00000000dfffffff 3.5G online yes 0-27
0x0000000100000000-0x000000021fffffff 4.5G online yes 32-67
Memory block size: 128M
Total online memory: 8G
Total offline memory: 0B
{"enabledMods": [
"lw_lazylib",
"MagicLib",
"shaderLib",
"dronelib",
"$$$_lightshow",
"su_Concord",
"lw_console",
"pantera_ANewLevel40R",
"aaacrew_replacer",
"Adjusted Sector",
"anotherportraitpack",
"portrAIt",
"raccoonarms",
"AttunedDriveField",
"automatedcommands",
"lw_autosave",
"beyondthesector",
"CaptainsLog",
"istl_dassaultmikoyan",
"edshipyard",
"fleetsizebydp",
"sun_flux_reticle",
"GrandColonies",
"HexShields",
"hte",
"hiigaran_descendants",
"immersionFriendlyPortraitPack",
"IndEvo",
"interestingportraitspack",
"Interstellar Federation Refurbished",
"lights_out",
"missingships",
"su_CarrierHullmod",
"sun_new_beginnings",
"nexerelin",
"scan_those_gates",
"console_overlord_additionalcommands",
"clearCommands",
"mkportraits",
"personal_portratis",
"pt_qolpack",
"HMI",
"Neutrino",
"forge_production",
"more_hullmods",
"wyv_planetaryShieldAccessControl",
"scf",
"planet_search",
"QualityCaptains",
"ArkLeg",
"swp",
"speedUp",
"stelnet",
"Terraforming & Station Construction",
"diyplanets",
"transpoffder",
"US",
"whichmod",
"audio_plus",
"tahlan",
"officerExtension",
"harmless-slipstreams",
"wisp_NeutrinoDetectorMkII",
"progressiveSMods",
"timid_admins",
"mayu_specialupgrades",
"armaa",
"exshippack",
"exshippack_armaacompat",
"mySettings"
]}