It's... a bit involved. Just gave it a whirl (with some good results). On a high level, what you want to do is connect to the game using some kind of Java profiling tool, do whatever you suspect causes a leak, and then take a memory snapshot (called a "heap dump") to see what's leaking and where.
Edit (8/3/2020): you can download jvisualvm
here.
One way to do this is:
1) Run jvisualvm.exe (found in the bin directory of a JDK installation)
2) Connect to StarfarerLauncher - now you have all sorts of neat real-time data available, including memory and processor usage.
3) In the game, do whatever it is you think may cause a memory leak (in this case, just reload a save)
4) In jvisualvm, right-click on the application and select "Heap Dump"
5) Open up the heap dump in the tab on the right - this is going to have the contents of the memory at the time the dump was taken
6) Sort the heap dump by the "classes" field, look for com.fs.stafarer.campaign.CampaignEngine
7) If the "Instances" count is more than 1, you've got a memory leak (there are other ways to have memory leaks, of course, but when you're leaking whole copies of the game world, that's generally a problem)
8
) Right-click on CampaignEngine, select "show in instances view"
9) There should be a couple of those visible in the "Instances" tab (#1, #2, etc). The last one is generally the real/current one, the other ones are leaks
10) Select one of those, then under "References", right-click on "this" and select "Show nearest GC root".
11) This is where it gets a bit confusing. It'll show you the thing that's holding a reference to the CampaignEngine instance and is preventing it from being garbage-collected, but it generally won't be something immediately recognizable. Go up the tree until you see something familiar (i.e. a class that's part of the mod), and see what field name is pointed out there. That's the likely source of the leak.
So, the end result of this process, for me, pointed to TheNomadsCloningVats hullmod, but the field was "memo", which wasn't present in the class. Looking in its base class - BaseFleetEffectHullMod - yielded this:
public abstract class BaseFleetEffectHullMod extends BaseHullMod
{
// possible long-term memory leak
private Hashtable memo = new Hashtable();
The comment's not mine, and yeah, it's a memory leak alright, for any hullmod using that base class. HullModEffects get instantiated once per application run, and shouldn't hold on to any data long term. You might be wondering why holding on to a fleet member inside "memo" is causing the whole CampaignEngine to not be garbage-collected - it's because there are internal references to the engine a few levels down from stuff in the fleet.
Important note: this does not mean this is the only memory leak. If I had to guess, I would guess there are others, since this type of mistake is easy to make (and it can be hard to know you're making it, for example it's not 100% clear exactly when HullModEffects get instantiated). So, basically, the process for hunting this type of thing down is doing these steps and fixing the problems one at a time until the instance count for CampaignEngine stays at 1 when you do the sequence of steps that used to cause problems.
Let me know if the above makes any sense, and if you've got any questions about it! It might seem intimidating, weighing in at 11 steps and all, but it's actually really fast to step through once you've got the hang of it - it literally only took me a minute or two, once I was all set up with the latest versions of the mods.
