Fractal Softworks Forum

Please login or register.

Login with username, password and session length
Advanced search  

News:

Starsector 0.97a is out! (02/02/24); New blog post: Simulator Enhancements (03/13/24)

Pages: 1 ... 3 4 [5] 6 7

Author Topic: [Late Game] Battles & Colony Management ruin FPS gradually & permanently.  (Read 16965 times)

SaberCherry

  • Commander
  • ***
  • Posts: 109
    • View Profile
Re: [Late Game] Battles & Colony Management ruin FPS gradually & permanently.
« Reply #60 on: October 09, 2020, 09:35:10 PM »

I'm not sure how relevant this is, but the one time I had a massive permanent slowdown - which was solved by saving (which took a long time), exiting, and reloading - was when I put a whole bunch of stuff into storage.  The game was fine; I pushed a bunch of stuff into storage, and while doing that, it got extremely sluggish...  and never recovered.  Since people mention this sometimes happening after a big battle or visiting a colony, I wonder if inventory management could be involved.

This was a modded game with Nexerelin and various others...  specifically:

{"enabledMods": [
  "raccoonarms",
  "automatic-orders",
  "lw_autosave",
  "alportraitpack",
  "alibportraitpack",
  "alrnportraitpack",
  "blackrock_driveyards",
  "CombatAnalytics",
  "DisassembleReassemble",
  "Imperium",
  "lw_lazylib",
  "ArkLeg",
  "luddenhance",
  "MagicLib",
  "nexerelin",
  "ORA",
  "TAR",
  "SCY",
  "shadow_ships",
  "swp",
  "speedUp",
  "Sylphon_RnD",
  "THI",
  "transfer_all_items",
  "US",
  "lw_version_checker",
  "yrxp",
  "shaderLib",
  "prv"
]}
« Last Edit: October 09, 2020, 09:41:09 PM by SaberCherry »
Logged

Jaghaimo

  • Admiral
  • *****
  • Posts: 661
    • View Profile
Re: [Late Game] Battles & Colony Management ruin FPS gradually & permanently.
« Reply #61 on: October 19, 2020, 02:17:01 AM »

Profiling continues to turn up nothing of interest, and console still indicates memory use is a fraction of the available amount.

...

I have the inkling that the slowdown stems from some unknown issue in the JVM.
Did you look at the distribution of free space inside heap space? We might be fine in eden and survivor space but running out of tenured space (or worse - permanent space).
My bet would be on Stop The World event caused by GC when it runs a major collection on tenured space.
(IIRC one of - not the only, there was some weird hard crashing, too - the issues with 8 is that its GC settings are by default different and can cause actual frame drops especially with larger allocations. There's, again iirc, some set of JVM parameters that could be added to use the default Java 7 GC behavior.)
Java7 uses serial collection, while Java8 uses parallel (serial if you use 32bit or single CPU machine). Neither seems good for heavily modded game with large heap sizes. Perhaps we should try running with G1 instead, as that's optimal for large memory (5-6G or larger) and low pause time (0.5s or less)?

https://www.oracle.com/technical-resources/articles/java/g1gc.html
« Last Edit: October 19, 2020, 02:25:18 AM by Jaghaimo »
Logged

HySOqvs0sB8

  • Ensign
  • *
  • Posts: 8
    • View Profile
Re: [Late Game] Battles & Colony Management ruin FPS gradually & permanently.
« Reply #62 on: October 19, 2020, 11:56:09 AM »

...

The game uses parallelGC by default, but the algorithm does not even matter here. I would expect the users to get an OOM error instead of gradually degrading FPS if the issue is memory related.
Logged

pairedeciseaux

  • Captain
  • ****
  • Posts: 340
    • View Profile
Re: [Late Game] Battles & Colony Management ruin FPS gradually & permanently.
« Reply #63 on: October 19, 2020, 06:51:16 PM »

Back on Java 7 (for testing purposes): I was able to reproduce in my game without Nexerelin or Tahlan, albeit with a significant number of other mods and I had to spawn + nuke a bunch of fleets with console to get a really noticeable effect.

Just to clarify,

You manually spawned one fleet using console, then fought the fleet. Then went to a market to repair own fleet? Then performed a number repetitions of the cycle? How many repetitions? Anything else? What was the approximate size of both fleets?

Would you consider that to be a proper reproducible test case at this point?

If yes, next steps could be: reproduce with vanilla-only, reproduce with small battles (if at all possible), have other people reproduce the slowdown given a simplified test case, pray profiling will show something.

The game uses parallelGC by default, but the algorithm does not even matter here. I would expect the users to get an OOM error instead of gradually degrading FPS if the issue is memory related.

If a significant amount of time is spent in GC, then GC tuning should be tried, no? I don't know, think fragmentation rather than leak, too much GC work to do, or other nasty JVM memory management behaviours.

Hmmm, I wonder whether tuning GC is a waste of time compared to things like: improving code in vanilla and mods when bottlenecks are identified, or, I guess, reducing game scope and scale ... but at the same time, my understanding is that no clear bottleneck has been found yet.

Does/can profiling used so far in this thread show unusual GC activity? It may not be relevant at all, but I'll mention it anyway: has anyone tried something like the following and checked the output log file?

Code
-Xloggc:../../../logs/gc.log -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps

I have run with theses options less than 30 seconds using an end-game vanilla save, simply moving own fleet from a system to hyperspace and gave a quick look at the output, see below. And to be clear, I did not try to reproduce the post-battle slowdowns discussed above, I am annoyed by unstable FPS both in-system and in hyperspace. Also please note I am not a Java person, at all.

Spoiler
Java HotSpot(TM) 64-Bit Server VM (24.79-b02) for bsd-amd64 JRE (1.7.0_79-b15), built on Apr 10 2015 11:35:04 by "java_re" with gcc 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.11.00)
Memory: 4k page, physical 41943040k(10485760k free)

/proc/meminfo:

CommandLine flags: -XX:+CompilerThreadHintNoPreempt -XX:CompilerThreadPriority=1 -XX:InitialHeapSize=4294967296 -XX:MaxHeapSize=4294967296 -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:ThreadStackSize=1024 -XX:+UseCompressedOops -XX:+UseParallelGC
4.988: [GC [PSYoungGen: 1049088K->25132K(1223680K)] 1049088K->25140K(4020224K), 0.0152840 secs] [Times: user=0.03 sys=0.00, real=0.02 secs]
7.505: [GC [PSYoungGen: 1074220K->81428K(1223680K)] 1074228K->81444K(4020224K), 0.0246230 secs] [Times: user=0.06 sys=0.03, real=0.02 secs]
10.627: [GC [PSYoungGen: 1130516K->164463K(1223680K)] 1131556K->165512K(4020224K), 0.0387260 secs] [Times: user=0.10 sys=0.05, real=0.04 secs]
12.305: [GC [PSYoungGen: 1213551K->78796K(1223680K)] 1218598K->83850K(4020224K), 0.0165760 secs] [Times: user=0.05 sys=0.00, real=0.02 secs]
13.447: [GC [PSYoungGen: 1127884K->50199K(1223680K)] 1132957K->55272K(4020224K), 0.0181630 secs] [Times: user=0.06 sys=0.00, real=0.02 secs]
19.294: [GC [PSYoungGen: 770431K->48056K(1240064K)] 775504K->53138K(4036608K), 0.0178920 secs] [Times: user=0.06 sys=0.00, real=0.02 secs]
19.312: [Full GC [PSYoungGen: 48056K->0K(1240064K)] [ParOldGen: 5081K->47048K(2796544K)] 53138K->47048K(4036608K) [PSPermGen: 25959K->25947K(52224K)], 0.1210730 secs] [Times: user=0.32 sys=0.01, real=0.12 secs]
22.974: [GC [PSYoungGen: 740456K->158181K(1207296K)] 787504K->237282K(4003840K), 0.0741000 secs] [Times: user=0.25 sys=0.02, real=0.07 secs]
23.048: [Full GC [PSYoungGen: 158181K->0K(1207296K)] [ParOldGen: 79100K->230314K(2796544K)] 237282K->230314K(4003840K) [PSPermGen: 32165K->32160K(63488K)], 0.3614440 secs] [Times: user=1.28 sys=0.03, real=0.36 secs]
Heap
 PSYoungGen      total 1207296K, used 646689K [0x00000007aaa80000, 0x0000000800000000, 0x0000000800000000)
  eden space 1049088K, 61% used [0x00000007aaa80000,0x00000007d2208708,0x00000007eab00000)
  from space 158208K, 0% used [0x00000007ec580000,0x00000007ec580000,0x00000007f6000000)
  to   space 163840K, 0% used [0x00000007f6000000,0x00000007f6000000,0x0000000800000000)
 ParOldGen       total 2796544K, used 230314K [0x00000006fff80000, 0x00000007aaa80000, 0x00000007aaa80000)
  object space 2796544K, 8% used [0x00000006fff80000,0x000000070e06a880,0x00000007aaa80000)
 PSPermGen       total 63488K, used 34326K [0x00000006fad80000, 0x00000006feb80000, 0x00000006fff80000)
  object space 63488K, 54% used [0x00000006fad80000,0x00000006fcf05ac8,0x00000006feb80000)
[close]
Logged

Histidine

  • Admiral
  • *****
  • Posts: 4661
    • View Profile
    • GitHub profile
Re: [Late Game] Battles & Colony Management ruin FPS gradually & permanently.
« Reply #64 on: October 20, 2020, 03:59:33 AM »

Back on Java 7 (for testing purposes): I was able to reproduce in my game without Nexerelin or Tahlan, albeit with a significant number of other mods and I had to spawn + nuke a bunch of fleets with console to get a really noticeable effect.

Just to clarify,

You manually spawned one fleet using console, then fought the fleet. Then went to a market to repair own fleet? Then performed a number repetitions of the cycle? How many repetitions? Anything else? What was the approximate size of both fleets?

Would you consider that to be a proper reproducible test case at this point?
I only tried it once, and don't really recall the sequence of events. There was no attempt at creating a reproducible test case, although I could try doing so using the save.

I think it was something like

Spawn fleet -> nuke -> do not return to a market -> spawn a new fleet and repeat

I think this came after one or more real battles, and I also did stuff like fly in hyperspace in an attempt to lower the framerate.
Spawnfleet command was used with (IIRC) 200 FP parameter. I think the actual fleets created by the command are larger than this.

@Jaghaimo: Given that I didn't understand any of that, this is probably my limit in terms of attempting to generate a hypothesis on the cause of the issue :-X
Logged

pairedeciseaux

  • Captain
  • ****
  • Posts: 340
    • View Profile
Re: [Late Game] Battles & Colony Management ruin FPS gradually & permanently.
« Reply #65 on: October 20, 2020, 03:22:24 PM »

Ok, running the spawnfleet and nuke commands was pretty easy. I tried both in a new vanilla campaign, and in my vanilla end-game save. I didn't obtain any permanent slowdown.

FYI here are launch count for commands I used in the new campaign, and of course I encountered all existing and naturally spawning Hegemony military fleets in Jangala + some pirates.

   1 spawnfleet tritachyon 50
   1 spawnfleet hegemony 100
  11 spawnfleet hegemony 200
  89 nuke


(depending on fleet size, between 1 and 3 nukes were necessary to defeat the fleet)

Maybe there wasn't enough battles. Or it can't be reproduced with vanilla-only. Or I don't know.

Please note my testing setup was different than my regular gaming setup. When I play the game, I use fullscreen 2560x1440 + sound. While testing this, I used windowed 1920x1080 + no sound. This is a LCD-type display with a 60 Hz refresh rate. Also I almost always use fully zoomed-out view. And I almost always use speed time mode.

That said,

I've made a few interesting observations looking at the fleet activity (and debris) + FPS + idle + CPU usage. Interesting to me, as these are probably obvious to people understanding the game's inner workings.

(I also quickly looked at GPU usage but didn't notice anything, Starsector only use a somewhat constant 50%)

CPU is a 7600K, so 4 cores at 3.8 GHz. Basically Starsector does most of its work on a single core, the percentages indicated below are single-core usage, unless I am badly mistaken.

Whether in a system or in hyperspace, there is a baseline CPU usage. In the new campaign, I had 30% inside Jangala system near the jump point, around 50% in hyperspace near the jump point, and around 60% near Askonia.

That CPU usage increases as fleets and debris are displayed. I got around +5% CPU usage per large fleet. And +10% more if mouse-over a fleet. This +5% increase per fleet seem really high.

So I guess the bigger the screen resolution, the bigger the number of potential displayed fleets and other objects, leading to higher CPU usage. Right? I haven't tried to isolate hyperspace storms CPU usage, but I guess these are kind of objects too and each have some CPU cost.

So I used batches of "spawnfleet player 200" and watched the result. This way I obtained easily reproducible "slowdowns". My understanding of what happened is:
  • a bunch of fleets are spawned at the same time
  • all are visible and displayed
  • they start moving doing their things
  • as long as all fleets are displayed, Starsector can't keep up with the 60 FPS goal
  • so Starsector drops to a 30 FPS goal, and I saw idle around 50%
  • as fleets leave the display area, Starsector changes it's goal back to 60 (or something higher than 30)

Now maybe I should not call these slowdowns, and I can't tell whether / how many people in this thread witnessed the same thing. I saw FPS instability. In the testing conditions, were the player fleet was static and the view stayed static too, it was only mildly annoying seeing the jerky fleet movements. But it is precisely this phenomenon that becomes very annoying once player fleet is moving and there is tons of objects and background display: you don't get a smooth scrolling anymore. Which definitely is something you want to avoid in a 2D game with a lot of scrolling, IMO.

Does the game also use an explicit 45 FPS goal or do you only encounter that when the game transition from 30 to 60, and from 60 to 30?

It seems the game is over-eager to go back to 60, and I suspect in many situations it only leads to bouncing back and forth from 30 to 60 to 30 to 60 ... with all the ugly intermediate FPS and non-smooth scrolling.

If this analysis is correct, I guess the right thing to do is:
  • ensure the game use either 30 or 60 (avoid intermediate)
  • use a "smart" condition to decide whether staying at 30 or upgrading to 60 such as (timeElapsedSinceLastChange > $DELAY) && (idle > $FPS_UPGRADE_REQ). Where $DELAY is something like 10 seconds, and $FPS_UPGRADE_REQ is something like 70%.
  • (check condition currently used to drop from 60 to 30)

If possible, reduce CPU cost to display a fleet?

Another thing to consider would be to implement a "level of detail" thing (as LOD in 3D applications with large scene and a lot of objects) where basically the game would only display the detailed fleet (each ship with its own engine trail) when zoomed-in (=pretty mode), and would display a simplified fleet (one ship or one abstract icon with a single trail) when zoomed-out (=performance mode). You get the idea.

Last minute tests: I checked and obtained the same behaviour fullscreen 2560x1440 with sound, also when turning off the speed time mode the problem is less likely to appear (I guess if game runs slower then it uses less CPU, so more CPU is available to handle peak load such as high number of fleets/objects to display). Which means there currently is two performance leverages: (1) use lower resolution in order to reduce the number of potential displayed objects (less CPU usage), (2) do not enable speed mode in order to "increase available CPU".
Logged

Alex

  • Administrator
  • Admiral
  • *****
  • Posts: 23986
    • View Profile
Re: [Late Game] Battles & Colony Management ruin FPS gradually & permanently.
« Reply #66 on: October 20, 2020, 03:29:44 PM »

Hmm - what you're describing is just normal "too much stuff on the screen" behavior, not any sort of bug or issue that can (or indeed needs to) be resolved.

Basically, past a certain number of things the game has to render, it won't be able to do it at 60 fps (this holds regardless of optimization; that point just comes later then), and so it'll drop to a lower frame rate. Since vsync is on, framerates lower than 60 will be numbers that divide evenly into 60 - so, 30, 20, 15, 12, 10, etc. Intermediate numbers are caused by the framerate changing from one of those numbers to the other, since the fps counter shows the average framerate for the last second.

The game will increase the timestep as the framerate drops - up to double at 30 fps - so there shouldn't be any slowdown until the game drops below 30.
Logged

pairedeciseaux

  • Captain
  • ****
  • Posts: 340
    • View Profile
Re: [Late Game] Battles & Colony Management ruin FPS gradually & permanently.
« Reply #67 on: October 21, 2020, 12:46:15 AM »

Hmm, sure one can't expect an infinite number of stuff rendered on screen smoothly.  :)

I am nonetheless surprised by the high CPU cost to render a single 200 FP fleet. Basically one could run several concurrent old classic 2D games (think shmups and run&guns with tons of bullets) or even old software rendered 3D games within that CPU budget (around a 150 MHz Pentium Pro), ... granted, at a lower resolution and lower color depth.

On my computer, ten 200 FP fleets displayed together is already too much in some areas. Whether that is acceptable or not is your call given intendended game scale and recommended computer specs.

I quickly tried with vsync OFF, and FPS was around 50; whereas with vsync ON FPS "goal" was jumping back and forth from 60 to 30 to 60, giving a similar choppy experience unless I am mistaken. One of the things I meant in previous message, maybe poorly phrased, is that experience with vsync ON would be smoother if the game stayed at 30 longer rather than instantly jumping back to 60. FPS instability is the main issue AFAIC, regardless of optimisations.

I suspect some players using tons of faction mods are experiencing these "FPS drops" because of the "high" number of displayed fleets (and possibly debris). I agree this is not a bug, more like an issue about pushing game scale + keeping a smooth rendering. Though I do wonder how this translates on slower computers with just the vanilla game in busy areas such as Askonia.

edit: fixed typo and added "smoothly" to first sentence
« Last Edit: October 21, 2020, 01:02:43 AM by pairedeciseaux »
Logged

Modo44

  • Commander
  • ***
  • Posts: 144
    • View Profile
Re: [Late Game] Battles & Colony Management ruin FPS gradually & permanently.
« Reply #68 on: October 21, 2020, 01:04:51 AM »

It's not the rendering that is costly, it's calculating all the object interactions. Sure it can be "optimized" by various hacks that calculate less, and emulate complexity with purty graphics, but that is not at the core of Starsector.

Have you noticed that most games only let you control battles in turns, or use very generalized control that only seems to affect every detail (e.g. you control the flags, not individual fighters in the Total War series). Now you know why.
Logged

Alex

  • Administrator
  • Admiral
  • *****
  • Posts: 23986
    • View Profile
Re: [Late Game] Battles & Colony Management ruin FPS gradually & permanently.
« Reply #69 on: October 21, 2020, 09:37:34 AM »

Ah, my main point is that this looks like a completely separate issue from what's discussed in the thread.
Logged

pairedeciseaux

  • Captain
  • ****
  • Posts: 340
    • View Profile
Re: [Late Game] Battles & Colony Management ruin FPS gradually & permanently.
« Reply #70 on: October 21, 2020, 11:13:13 AM »

It's not the rendering that is costly, it's calculating all the object interactions.

I'm not 100% sure, but it really looks like slow rendering: no fleet on sight = 40% CPU, large fleet enters screen = 45% CPU, large fleet leaves screen = back to 40% CPU.

If the cost of "calculating all the object interactions" were that high for each fleet, the game would be unplayable.

Ah, my main point is that this looks like a completely separate issue from what's discussed in the thread.

Fair enough. I came to this from a vaguely-related-consequences point of view (FPS drops) and wasn't sure whether to post here or elsewhere. Feel free to move or delete those messages.
Logged

Lance

  • Ensign
  • *
  • Posts: 12
    • View Profile
Re: [Late Game] Battles & Colony Management ruin FPS gradually & permanently.
« Reply #71 on: October 27, 2020, 10:13:33 AM »

Hello, i registered to say that i have/had the same problem.

After hours of simulations & tests in the -REFIT- screen of the ships, with different weapons and ships, the game was slowd permanently, even without playing the campain, doing real fights or touching the command screen.


It somehow got better when i removed Speedup Mod and put the combat speed at *2 in the game settings instead. The game ran way better after a lot of fights, be it simulated or not.  Maybe static bloat?  anyway, i'm sure that speed up was the main culprit for this specific problem.


 Another thing that slows the game drastically is the amount of weapons. Or something tied to it:

Thoses weapons, in your home's storage, could slow the game whenever a techmined item is added, and whenever we access the command panel? Nowaday i send the techmined items on another planet than my home and it seems better...  (i mean, my official storage is not my home, more accurate) but it could simply be pure luck... Note, i'm talking about 5000++ weapons and atleast 100+ different ones. ( not 5000*vulcan cannon but 100 units of vulcan/devastator/markix/..... etc)

a third thing that slows the performances massively, it's the stations in hyperspace, with patrol hq/command center, they can be added by tahlan and vayra sector

Also, vayra sector, on a huge modded game, reduces my fps from 60 to 45 in campaign with a FRESH & new game. So i believe it is the main culprit for bad performances. It's a shame because HVB are amazing...
for java 8, it stutters a bit more and average fps is a bit worse...  i dunno about the long term performance
« Last Edit: October 28, 2020, 04:03:31 AM by Lance »
Logged

Drglord

  • Lieutenant
  • **
  • Posts: 79
    • View Profile
Re: [Late Game] Battles & Colony Management ruin FPS gradually & permanently.
« Reply #72 on: December 16, 2020, 09:52:21 AM »

I'm curious in knowing if anyone that has been getting these slowdowns can report through the task manager if java is using more than 3G. I have a feeling that the game wants to use more memory but it can't for whatever reason, even when set to.

My version uses up to 7GB of memory i have 8192 allocated. CPU usage is 12% and graphic maxes out in 50%. There is definetely something slowing it down but it is difficult to track down with so many mods.
Logged

Drglord

  • Lieutenant
  • **
  • Posts: 79
    • View Profile
Re: [Late Game] Battles & Colony Management ruin FPS gradually & permanently.
« Reply #73 on: December 16, 2020, 10:07:19 AM »

I think the space stations of legio infenalis drop down the perfomance ALOT that are in hyperspace. I have noticed when you are in a area with one a massive slow down occurs.
Logged

Flacman3000

  • Commander
  • ***
  • Posts: 202
  • A man cannot put his shaft into the unknown.
    • View Profile
Re: [Late Game] Battles & Colony Management ruin FPS gradually & permanently.
« Reply #74 on: January 02, 2021, 05:07:11 PM »

did anyone find a solution to this situation so many memory leaks but I have all the mods we narrowed down for the sake of improved enjoyment. It's sad and unfortunate is there a way to change the java that's running the game id like to try java 8 but I don't know how I cant see what java to replace in star sector core or what to do in vmparam.
Logged
Ryzen 7 2700x, 1080ti, 16GB DDR4 RAM and playing SS on a SSD - Always stay ahead because when you're ahead you're rear anatomy is open to interpretation.
Pages: 1 ... 3 4 [5] 6 7