Fractal Softworks Forum

Please login or register.

Login with username, password and session length
Advanced search  

News:

Starsector 0.97a is out! (02/02/24); New blog post: Simulator Enhancements (03/13/24)

Author Topic: 0.9.1a weapon testing - Thumper experiment  (Read 3189 times)

pairedeciseaux

  • Captain
  • ****
  • Posts: 340
    • View Profile
0.9.1a weapon testing - Thumper experiment
« on: July 12, 2020, 03:51:36 PM »

(posted on 2020-07-13, then edited on 2020-07-18 to adjust the TL;DR and fix a typo)

So I wanted to evaluate the usefullness of the Thumper medium ballistic weapon, mainly to learn about the testing methodology and tools, less so for the insight about Thumper.

I suspected it would be hard to gain knowledge on Thumper using repetitive tests while keeping a realistic simulation, but still wanted to try. What I mean by realistic is I don't want to perform micro-benchmarks measuring just the effect of the weapon on a dumb target, and ideally I want to recreate conditions that exists in real campaigns. But whether a success or failure, it would serve as a good preperation for more tests.

TL;DR - Overall replacing 1 Flak by 1 Thumper on Enforcer did not bring benefit in these tests, either because it really does not, or because the test failed to measure it.  ;D  Update: You can safely stop reading here unless you are interested in the testing process or subtle fleet battle issues. The first experiment was really flawed, I got better measurements in my second attempt, details in another message below. Short version is: Heavy Mortar has better performance than Thumper in these tests.

Testing effort inspired by: https://fractalsoftworks.com/forum/index.php?topic=18804

and: https://fractalsoftworks.com/forum/index.php?topic=15758

Disclaimer

This is my first attempt at repetitive Starsector fleet testing and I'm no pro statistician. Don't put too much faith in data or statistics shown below.

Ships and loadout

I used Enforcers as a test platform, with a custom loadout, from left to right: 1 Flak, 1 Arbalest, 1 Heavy Mortar, 1 Arbalest, 1 Flak. And also 4x2 Hammers in front, Reinforced Bulkhead, Flux Distributor and Auxiliary Thrusters. This is the reference variant. I also tested a variant where I swapped the right-side Flak for a Thumper.

On the other side, I used Condors, the Attack variant: 2 Broadsword wings, 1 Pillum, 2 Vulcan. This is the reference variant. I also used one variant without any fighter, and another variant without any missile.

All fights had 2 Enforcers against 2 Condors, all with aggressive personnality.

Method

I mesured the time to destroy Condors accross all matchups, and did 18 repetitions of each. I didn't use results when Condor retreated. Statistic analysis gave a few data points, including median time and time distribution accoss 1 minute wide segments (number of samples per segment).

Starsector 0.9.1a-RC8, by FractalSoftworks
Fleet Tester mod 1.0, by Tartiflette
MagicLib 0.28, by Tartiflette
LazyLib 2.4e or f, by LazyWizard

Running in 1024x768 in windowed mode, with sound disabled.
I ran 3 simultaneous battles in 3 Starsector instances, each running in real time (no speed increase).
So 3 windows in the top area of the screen, with a slight overlap, looking at the 3 battle map.
And 3 terminal windows, manually starting/stopping "/usr/bin/time sleep 600".
Manually writing times in a speadsheet. Speadsheet also used to perform statistic analysis.

Results

Against 2 Condors with 2 Broadswords and 1 Pillum
2 Enforcers with 2 Flaks are slightly faster than 2 Enforcers with 1 Flak and 1 Thumper
the median time difference is around 7%, 191 vs 204
time distribution:
2 to 3 minutes: 8 vs 7
3 to 4 minutes: 7 vs 7
4 to 5 minutes: 1 vs 2
over 5 minutes: 2 vs 2

Against 2 Condors with 1 Pillum (no fighters)
2 Enforcers with 2 Flaks are slightly faster than 2 Enforcers with 1 Flak and 1 Thumper
the median time difference is around 12%, 98 vs 110
time distribution:
1 to 2 minutes: 16 vs 15
2 to 3 minutes: 2 vs 3

Against 2 Condors with 2 Broadswords (no Pillum)
2 Enforcers with 2 Flaks are slightly slower than 2 Enforcers with 1 Flak and 1 Thumper
the median time difference is around 8%, 227 vs 209
time distribution:
2 to 3 minutes: 4 vs 4
3 to 4 minutes: 6 vs 9
4 to 5 minutes: 6 vs 3
over 5 minutes: 2 vs 2

An additial comparison worth looking at, with 2 Enforcers with 2 Flaks
2 Condors with 2 Broadswords and 1 Pillum is faster (at being destroyed) than 2 Condors with 2 Broadswords (no Pillum)
the median time difference is around 16%, 191 vs 227
2 to 3 minutes: 8 vs 4
3 to 4 minutes: 7 vs 6
4 to 5 minutes: 1 vs 6
over 5 minutes: 2 vs 2

Comments

I'm not 100% convinced about the sample size. I got 12 initialy, then moved to 18: it did not change the time distribution very much, it did not change the faster/slower situation. So it looks good, but I'm not sure. ;-)

I wrote above that I measured the time to destroy Condors. What happened in the simulation was more like:
time spent to travel to Condors (offense)
time spent to deal with fighters (defense)
time spent to deal with missiles (defense)
time spent to vent (offense and defense)
time spent to deal actual damage to Condors (offense)

I don't have the breakdown, but we can see the no fighters tests were twice as fast as the tests with fighters.

So what I tried to measure was only a small part of the actual measurement. Signal to noise ratio not that good. Not really a surprise, but a better test design could help.

In the end those tests showed Enforcer behaviour variation depending on external pressure. In other words, I tested defense againts this pressure as much as offense against the target ships. Which is not a bad thing to do with respect to the Thumper: this weapon should provide value against both fighters and ships. But doing it in the same test is hard IMO. Testing defense could be done separately - such as measuring time to loose against a swarm of fighters, carrier being impossible to catch.

Speaking about offense and defense, Enforcers shot Hammers a lot against both Broadswords and Condors. IMO, this reduced my ability to measure differences between 2 Flaks and 1 Flak + 1 Thumper. But I didn't want to run a test with a missile-less Enforcer because it is not realistic / not my preference (well, it's a player choice, so it could be), so I'm still confortable with that choice. Still... it might be worth checking in another test setup.

The Broadswords sure did put a lot of pressure on the Enforcers. The temporary stunlock effect is quite impressive. It is somewhat representative of what a fighter heavy fleet would impose on its opponent (Legion with 4 Broadsword wings would do this). I added the no fighters matchups to check results without this pressure. Interesting to see the result difference ... and similitude.

The Pillum situation is less clear.

With Pillum pressure I got shorter time than without Pillum pressure. I've got 2 mad theories on this. (1) Enforcer with dual Flaks got improved averall DPS while damaging missiles and fighters simultaneouly. Hmm, that does not sound right. (2) Condors without Pillum where less compelled to stay in sensor range, so could kite from a longer distance. Hmmm (3) or results are just plain wrong.

Tentative conclusion

Did 1 Thumper on Enforcer provide measurable benefits in the no fighter tests? No it was worse according to statistics.

Did 1 Thumper on Enforcer provide measurable benefits in the tests with Broadswords? Yes in 1 case (no Pillum) and no (it was worse) in 1 case (Pillum), according to statistics. So no expected overall benefits considering mixed real campaign situations.

If it is so difficult with Enforcer, my brain hurts trying to imagine testing effect of having 1 or 2 Thumpers on Onslaugth or Legion with realistic loadout (both being candidate because they have plenty medium ballistic turrets).

Comments and questions welcome! Also having more than one pair of eyes checking results and methodology sure is usefull.

Data

player0_fleet.csv
Spoiler
rowNumber,variant,personality,flagship
1,enforcer_Custom_FlakAndThumper,aggressive,false
2,enforcer_Custom_FlakAndThumper,aggressive,false
[close]

player1_fleet.csv
Spoiler
rowNumber,variant,personality,flagship
1,condor_Attack_Custom_NoPillum,aggressive,false
2,condor_Attack_Custom_NoPillum,aggressive,false
[close]

enforcer_Custom_FlakOnly.variant
Spoiler
{
    "displayName": "Custom FlakOnly",
    "fluxCapacitors": 15,
    "fluxVents": 20,
    "goalVariant": true,
    "hullId": "enforcer",
    "hullMods": [
        "reinforcedhull",
        "fluxdistributor",
        "auxiliarythrusters"
    ],
    "permaMods": [],
    "quality": 0.5,
    "variantId": "enforcer_Custom_FlakOnly",
    "weaponGroups": [
        {
            "autofire": false,
            "mode": "ALTERNATING",
            "weapons": {
                "WS 006": "hammer",
                "WS 007": "hammer",
                "WS 008": "hammer",
                "WS 009": "hammer"
            }
        },
        {
            "autofire": true,
            "mode": "LINKED",
            "weapons": {"WS 003": "heavymortar"}
        },
        {
            "autofire": true,
            "mode": "LINKED",
            "weapons": {
                "WS 001": "flak",
                "WS 005": "flak"
            }
        },
        {
            "autofire": true,
            "mode": "LINKED",
            "weapons": {
                "WS 002": "arbalest",
                "WS 004": "arbalest"
            }
        }
    ]
}
[close]

enforcer_Custom_FlakAndThumper.variant
Spoiler
{
    "displayName": "Custom FlakAndThumper",
    "fluxCapacitors": 14,
    "fluxVents": 20,
    "goalVariant": true,
    "hullId": "enforcer",
    "hullMods": [
        "reinforcedhull",
        "fluxdistributor",
        "auxiliarythrusters"
    ],
    "permaMods": [],
    "quality": 0.5,
    "variantId": "enforcer_Custom_FlakAndThumper",
    "weaponGroups": [
        {
            "autofire": false,
            "mode": "ALTERNATING",
            "weapons": {
                "WS 006": "hammer",
                "WS 007": "hammer",
                "WS 008": "hammer",
                "WS 009": "hammer"
            }
        },
        {
            "autofire": true,
            "mode": "LINKED",
            "weapons": {"WS 003": "heavymortar"}
        },
        {
            "autofire": true,
            "mode": "LINKED",
            "weapons": {
                "WS 001": "flak"
            }
        },
        {
            "autofire": true,
            "mode": "LINKED",
            "weapons": {
                "WS 005": "shredder"
            }
        },
        {
            "autofire": true,
            "mode": "LINKED",
            "weapons": {
                "WS 002": "arbalest",
                "WS 004": "arbalest"
            }
        }
    ]
}
[close]

condor_Attack_Custom_NoFighter.variant
Spoiler
{
    "displayName": "CustomNoFighters",
    "fluxCapacitors": 7,
    "fluxVents": 7,
    "goalVariant": true,
    "hullId": "condor",
    "hullMods": [],
    "permaMods": [],
    "variantId": "condor_Attack_Custom_NoFighter",
    "weaponGroups": [
        {
            "autofire": false,
            "mode": "ALTERNATING",
            "weapons": {"WS 003": "pilum"}
        },
        {
            "autofire": true,
            "mode": "LINKED",
            "weapons": {
                "WS 001": "vulcan",
                "WS 002": "vulcan"
            }
        }
    ],
    "wings": [
    ]
}
[close]

condor_Attack_Custom_NoPillum.variant
Spoiler
{
    "displayName": "CustomNoPillum",
    "fluxCapacitors": 7,
    "fluxVents": 7,
    "goalVariant": true,
    "hullId": "condor",
    "hullMods": [],
    "permaMods": [],
    "variantId": "condor_Attack_Custom_NoPillum",
    "weaponGroups": [
        {
            "autofire": true,
            "mode": "LINKED",
            "weapons": {
                "WS 001": "vulcan",
                "WS 002": "vulcan"
            }
        }
    ],
    "wings": [
        "broadsword_wing",
        "broadsword_wing"
    ]
}
[close]

Matchups CSV
Spoiler
;2 Enforcers with 2 Flaks;2 Enforcers with 1 Flak and 1 Thumper
2 Condors with 2 Broadswords and 1 Pillum;set 1A;set 1B
2 Condors with 1 Pillum (no fighters);set 2A;set 2B
2 Condors with 2 Broadswords (no Pillum);set 3A;set 3B
[close]

Raw data CSV
Spoiler
;set 1A;set 1B;empty;set 2A;set 2B;empty;set 3A;set 3B
1;225;241;;99;99;;226;200
2;162;177;;97;119;;170;212
3;390;212;;120;130;;228;320
4;197;196;;96;127;;259;241
5;223;215;;85;113;;211;195
6;166;306;;94;110;;197;130
7;206;131;;113;101;;288;213
8;135;320;;100;97;;244;261
9;215;175;;105;89;;167;151
10;136;221;;89;167;;263;190
11;314;269;;83;110;;347;208
12;172;157;;91;112;;164;179
13;136;150;;130;103;;376;303
14;171;185;;89;101;;219;209
15;185;142;;140;111;;192;289
16;262;231;;117;94;;151;216
17;214;227;;84;77;;283;180
18;133;156;;105;120;;264;199
[close]

Statistical analysis CSV
Spoiler
;set 1A;set 1B;empty;set 2A;set 2B;empty;set 3A;set 3B;interval values;interval indexes
min;133;131;;83;77;;151;130;;
max;390;320;;140;167;;376;320;;
average;202;206;;102;110;;236;216;;
median;191;204;;98;110;;227;209;;
max - min;257;189;;57;90;;225;190;;
stddev;67;54;;16;19;;62;50;;
avedev;48;43;;13;13;;48;37;;
freq 0 to 1 min;0;0;;0;0;;0;0;60;1
freq 1 to 2 min;0;0;;16;15;;0;0;120;2
freq 2 to 3 min;8;7;;2;3;;4;4;180;3
freq 3 to 4 min;7;7;;0;0;;6;9;240;4
freq 4 to 5 min;1;2;;0;0;;6;3;300;5
freq over 5 min;2;2;;0;0;;2;2;;6
[close]
« Last Edit: July 18, 2020, 10:27:37 AM by pairedeciseaux »
Logged

RustyCabbage

  • Captain
  • ****
  • Posts: 347
    • View Profile
Re: 0.9.1a weapon testing - Thumper experiment
« Reply #1 on: July 14, 2020, 01:53:42 PM »

Thanks for giving this a shot. More methodological testing is a good thing to see.

Some issues:
1) Time-to-kill is such a high variance measure that it's extremely difficult to tell whether a given result is due to switching the weapons around or simple RNG from the fight. It'd be less of a problem with way more samples and a greater effect size, but
2) The potential effect of replacing a single weapon is probably too small to be distinguishable from noise unless you massively scale the battle which introduces additional variance anyways. The effect of the Hammers alone probably skew the results beyond reasonable inference.
3) By excluding the battles with retreating Condors, you're assuming that they retreat randomly in a way that does not affect the overall distribution, rather than as a function of how the battle was going. I don't know if Fleet Tester has a no-retreat option, but if not I think AI Battles (from the Discord) is probably going to be more reliable for testing.
4) Does Fleet Tester allow varying battlespaces? Which one you're using is also good to mention if it has that functionality.

I guess you noted some of these in your analysis, but they bear repeating.

Thoughts on a better design:
1) Remove the Hammers. You can leave the 8 OP empty if you want to pretend they're there, but measuring Thumper performance beside them is like trying to measure the brightness of a lightbulb outdoors on a sunny day.
2) Run more Thumpers. The Enforcer is a good platform for it, but you'd need to 2-3 of them over more ships to have a discernable effect.
3) From what I understand Thumper supporters think of it more as a frigate/light destroyer melter than anti-fighter. Perhaps do more trials against frigates of varying armor/shield strength and as a replacement for the Heavy Mortar rather than the Flak.

pairedeciseaux

  • Captain
  • ****
  • Posts: 340
    • View Profile
Re: 0.9.1a weapon testing - Thumper experiment
« Reply #2 on: July 14, 2020, 04:29:28 PM »

Thank you RustyCabbage for the review and for sharing your thoughts on improving the test design.

I agree with most points you wrote. Basically, yes, my test was designed to fail with only 1 Thumper on Enforcer, 8 Hammers, fighter pressure, and so on. Good news is I've now done another test where I've been able to reuse some of the design with better success: medium gun on Wolf.

3) By excluding the battles with retreating Condors, you're assuming that they retreat randomly in a way that does not affect the overall distribution, rather than as a function of how the battle was going.

I measured time to destroy Condor. So Condor retreating is invalid data as far as the test is concerned. Does it make sense?

4) Does Fleet Tester allow varying battlespaces? Which one you're using is also good to mention if it has that functionality.

I have no idea about configuration beyond the fleet composition at this point.  :) I'm using the version available on the forum. Indeed "mapsize" being quite small was annoying in some situations, but beneficial in others. And all to those capture points introduced more noise to the measurements.

Thoughts on a better design:
1) Remove the Hammers. You can leave the 8 OP empty if you want to pretend they're there, but measuring Thumper performance beside them is like trying to measure the brightness of a lightbulb outdoors on a sunny day.

Right.  ;D  On the other hand an Enforcer without missiles is heresy.  ;)

There is a conflict between the desire to have a successful test and the desire to create a realistic simulation. That's annoying, I'm torn about this.

2) Run more Thumpers. The Enforcer is a good platform for it, but you'd need to 2-3 of them over more ships to have a discernable effect.

Agreed. Thought about this, as a special escort-type loadout for Enforcer, maybe testing full Flak vs mixed Flak + Thumper. As a player I would still prefer my general purpose loadout in all cases.

3) From what I understand Thumper supporters think of it more as a frigate/light destroyer melter than anti-fighter. Perhaps do more trials against frigates of varying armor/shield strength and as a replacement for the Heavy Mortar rather than the Flak.

It's a good idea.

It's not practical with Enforcer as platform because it would have a hard time catching frigates, so you would measure time to run behind frigates more than anything else.

What is left as platform candidates is a few frigates with medium ballistic slots. I did not give enough thoughs to this approach because I don't like the idea of not having a kinetic and/or high explosive gun on those ships, but maybe it's just me.
Logged

Thaago

  • Global Moderator
  • Admiral
  • *****
  • Posts: 7174
  • Harpoon Affectionado
    • View Profile
Re: 0.9.1a weapon testing - Thumper experiment
« Reply #3 on: July 14, 2020, 04:44:45 PM »

More testing is great! Thanks for laying out your methodology so clearly, its good to see. I think overall battle result, along with time, is probably the best metric to be judging here, so if a Condor retreats thats ok because its overall battle time being measured, rather than time to kill.

In terms of platform, you could also try a Hammerhead with Thumpers in the medium slots.
Logged

RustyCabbage

  • Captain
  • ****
  • Posts: 347
    • View Profile
Re: 0.9.1a weapon testing - Thumper experiment
« Reply #4 on: July 14, 2020, 06:45:34 PM »

I measured time to destroy Condor. So Condor retreating is invalid data as far as the test is concerned. Does it make sense?
Sure, but it's an issue because you don't know if the retreat scenarios are completely at random. Suppose the Condors only manage to retreat if they would otherwise die in sub-120 seconds. Then removing any points where retreats happen will skew your data and give an incomplete picture. Unless you can justify saying that retreats happen randomly you can't merely disregard them. Better to just use AI Battles or something which doesn't allow the AI to retreat.

It's a good idea.

It's not practical with Enforcer as platform because it would have a hard time catching frigates, so you would measure time to run behind frigates more than anything else.

What is left as platform candidates is a few frigates with medium ballistic slots. I did not give enough thoughs to this approach because I don't like the idea of not having a kinetic and/or high explosive gun on those ships, but maybe it's just me.
Yeah, unless you change the metric to be something like battle result (maybe in conjunction with time as Thaago suggests), using the Enforcer is pretty rough. You also probably have to remove missiles from the frigates as well (really great seeing Enforcers lose a 5v5 against mere Lashers).

In terms of platform, you could also try a Hammerhead with Thumpers in the medium slots.
One issue is that (at least according to the wiki) the Thumper apparently doesn't benefit from Accelerated Ammo Feeder. It would otherwise seem pretty good though. Maybe the Brawler is an option with 1 Thumper and 1 medium kinetic?

Thaago

  • Global Moderator
  • Admiral
  • *****
  • Posts: 7174
  • Harpoon Affectionado
    • View Profile
Re: 0.9.1a weapon testing - Thumper experiment
« Reply #5 on: July 14, 2020, 07:53:29 PM »

...
In terms of platform, you could also try a Hammerhead with Thumpers in the medium slots.
One issue is that (at least according to the wiki) the Thumper apparently doesn't benefit from Accelerated Ammo Feeder. It would otherwise seem pretty good though. Maybe the Brawler is an option with 1 Thumper and 1 medium kinetic?

Oh, I had no idea! Is it a fire rate thing? Wait, does AAF work with vulcans? My mind is blown.
Logged

Alex

  • Administrator
  • Admiral
  • *****
  • Posts: 23988
    • View Profile
Re: 0.9.1a weapon testing - Thumper experiment
« Reply #6 on: July 14, 2020, 09:28:31 PM »

The maximum rate of fire is 20 rounds per second! This is governed by minRefireDelay in settings.json. I forget what the RoF of the Thumper is in the burst - might be 20/second? but it definitely benefits from the cooldown in between bursts being halved. The Vulcan, on the other hand, doesn't benefit at all.
Logged

SCC

  • Admiral
  • *****
  • Posts: 4112
    • View Profile
Re: 0.9.1a weapon testing - Thumper experiment
« Reply #7 on: July 15, 2020, 12:48:44 AM »

I'd rather test Enforcer with 1 HAC, 1 Heavy Mortar and 1 Thumper (and other guns and stuff, obviously) vs 2 HAC, 1 Heavy Mortar one (or 1 HAC, 2 Heavy Mortar, whatever's your preference). It's not really important if Thumper can perform in a vacuum, but rather if it gives enough benefit to swap one other weapon for it. This also limits it to Enforcer, Onslaught and maybe Legion, because everything else can't really spare the mounts.

Grievous69

  • Admiral
  • *****
  • Posts: 2980
    • View Profile
Re: 0.9.1a weapon testing - Thumper experiment
« Reply #8 on: July 15, 2020, 01:08:50 AM »

I know this isn't anywhere near the level of tests pairedeciseaux has been doing but since Thaago wanted to know how it performed on Hammerheads, I was also a bit curious to see it. I just used the simulator and put two Balanced Hammerhead variants fight each other. The only difference is that my variant switched one Heavy Mortar for a Thumper (I had to remove Blast doors to free up OP) and the next one switched both for Thumpers (again with left out OP). The mixed one wasn't actually that bad, the flux levels on both ships kinda go the same way with each encounter so even hitting 0.8 shields with a Thumper doesn't make a huge difference. Now it started being obvious when shields were down, my Hammerhead was doing way less damage for a long time unless it decided to fire like 6 Harpoons and overload the other one. Then it could actually do some damage but in the end still lost due to Railguns + Harpoons, and it was turning like crazy (maybe it tried to spread out damage on armor idk).

Now the dual one, it's definitely far worse despite costing 4 OP more on guns. The only times it actually does something is when it spams Harpoons (12 of them to be exact since it has EMR). Now I kinda agree having missiles here kinda screws with the testing but since most of you said it would be unrealistic because of missiles being usually mounted, I guess the Balanced variants check all of that. So we're talking talking about a 500 armor ship, that's really not that much. I guess it could kill slower frigates fast but then again, even Railguns can finish them off (or fighters). I really don't see the niche here.

My conclusion: Never put Thumpers on a Hammerhead unless you have literally nothing else to mount (even small weapons).

Now there was one quite weird event I managed to capture with the Hammerheads:


At 80% flux with shields down, so clearly it's losing flux, it decided to completely back out and vent when the other Hammerhead was like at 2% hull. There were no other enemies around whatsoever, it had a clear shot at it. This is one of those things that happens from time to time but I always forget to capture it.
« Last Edit: July 15, 2020, 01:11:03 AM by Grievous69 »
Logged
Please don't take me too seriously.

Megas

  • Admiral
  • *****
  • Posts: 12118
    • View Profile
Re: 0.9.1a weapon testing - Thumper experiment
« Reply #9 on: July 15, 2020, 05:23:55 AM »

This also limits it to Enforcer, Onslaught and maybe Legion, because everything else can't really spare the mounts.
And Eagle, with its three hardpoints up front.  Maybe (starter) Hammerhead too if Railguns are in the smalls and Mortar and Thumper in the hardpoints.
Logged

pairedeciseaux

  • Captain
  • ****
  • Posts: 340
    • View Profile
Re: 0.9.1a weapon testing - Thumper experiment
« Reply #10 on: July 18, 2020, 10:08:25 AM »

Alright, I’ve run a few more tests using several of your ideas. I’ve also split the test campaign in two parts like I did on my “medium gun on Wolf” test campaign.

All tests were performed using 6 concurrent Starsector instances. I used the in game timer as measurement tool.

Part 1 - Hammerhead attacking a dumb target

Our test platform is a Hammerhead with 19 flux capacity, 20 flux vents, Flux Distributor, 2 Railguns, tested in 3 versions:
  • 2 Heavy Mortars, 20 spare OP
  • another version with the right side Heavy Mortar replaced by a Thumper, 18 spare OP
  • another version with the right side hardpoint left empty, 27 spare OP

The spare OP is here to pretend we have missiles, PD and missing weapon. We don’t use those in order to have a cleaner simulation focused on what we are interested in. Also we have a somewhat balanced attack loadout with either kinetic + high explosive or kinetic + high explosive + fragmentation.

Test fleet has only one such Hammerhead, with aggressive personality.

On the other side there is a single Condor used as a dumb target, tested in two versions:
  • strong: 700 arbor, 7000 hull, 6200 flux capacity
  • weak: 350 armor, 3500 hull, 2200 flux capacity

Condor has no weapon, no fighter, reckless personality. This also helps having a cleaner simulation.

So we have 6 matchups. I’ve run 24 repetitions of each, measuring the time to destroy the dumb target.

Against the weak target, from best to worst, the data point in curly brakets is the {total additional time (adding together all repetitions) compared to the baseline version}:
  • (#1) 1 Hammerhead with 2 Heavy Mortars, {-118 seconds}, median is 52
  • (#2) 1 Hammerhead with 1 Heavy Mortar and 1 Thumper, {-70 seconds}, median is 54
  • (#3) 1 Hammerhead with 1 Heavy Mortar, {0 second}, median is 56, total is 1353

Same against the strong target:
  • (#1) 1 Hammerhead with 2 Heavy Mortars, {-292 seconds}, median is 70
  • (#2) 1 Hammerhead with 1 Heavy Mortar and 1 Thumper, {-112 seconds}, median is 79
  • (#3) 1 Hammerhead with 1 Heavy Mortar, {0 second}, median is 83, total is 1981

I think we can firmly establish that, on Hammerhead attacking the tested targets:
  • 1 Thumper is better than 1 empty hardpoint
  • 1 Thumper is worse than 1 Heavy Mortar even when there is another Heavy Mortar stripping target’s armor

Part 2 - Enforcer defending from a swarm of fighters

Our test platform is an Enforcer with 15 flux capacity, 20 flux vents, Flux Distributor, Reinforced Bulkhead, Armored Weapon Mounts, 2 medium guns on the side mounts, and a lot of spare OP to account missing guns and missiles.

Using the side mounts means we have 360 degrees coverage of the tested weapon, with minimal overlap of the two guns. And each gun is in a dedicated group. So we are testing how effective is one such weapons at shooting down small targets.

Tested loadout were : 2 Thumpers, 2 Flaks, 2 Arbalests, 2 Heavy Mortars, an a no gun version. This is synthetic setup designed to evaluate performance of the Thumper compared to other low-end medium guns.

Test fleet has only one such Enforcer, with timid personality.

On the other side there are two Drovers each with 2 Gladius wings and 2 Light Autocannons, with timid personality. Gladius has 75 armor, 500 hull and good mobility.

So we have 5 matchups. I’ve run 24 repetitions of each, measuring the Enforcer survival time. The data points in curly brackets are the {total additional survival time, percentage of samples surviving longer than 180 seconds}. From best to worst:
  • (#1) 1 Enforcer with 2 Heavy Mortars, {1924 seconds, 100%}, median is 219
  • (#2) 1 Enforcer with 2 Thumpers, {1309 seconds, 63%}, median is 192
  • (#3) 1 Enforcer with 2 Flak, {987 seconds, 38%}, median is 179
  • (#4) 1 Enforcer with 2 Arbalest, {477 seconds, 0%), median is 159
  • (#5) 1 Enforcer with no weapon, {0 seconds, 0%}, median is 142, longer survival time is 178, total 3392

Also with Thumpers the time dispersion is higher than others, meaning less predictable/consistent performance.

I think we can firmly establish that against Gladius:
  • Thumper is worse than Heavy Mortar
  • Thumper is better than Flak and Arbalest

Tentative conclusion

Considering low-end medium ballistic guns to mount on frigates and destroyers… Thumper don’t shoot at missiles, so it cannot replace Flak. Obviously Thumper will perform worse than Arbalest against shield, and target ship will often lower shield to absorb Thumper’s burst on armor. Thumper don’t perform as well as Heavy Mortar (in these tests), but cost more OP. So it looks like the only situation one should use Thumper is when already having a Thumper but no Heavy Mortar available (while having both anti-missiles and anti-shield duties already covered by other guns).

Not tested here but worth considering: how would Thumper perform compared to some small ballistic gun, such as the Light Assault Gun? Ahem.

On cruisers and capitals, in theory it should be possible to find a good use for Thumper such as anti-fighter gun. But there it is still in competition with Flak, Dual Flak, various kinetic and high explosive options ... and also missiles on medium composite mounts. I haven't tried, but currently I can't imagine using a Thumper on Falcon, Eagle, Dominator. No way either on Gryphon, Heron and Mora. Maybe 1 Thumper could prove to be a decent option on Legion's central medium mount.

Comments

Well, testing Thumper did not change my assessment of the weapon itself. But it sure helped me learn about the testing process. Thank you guys for giving me useful pointers.

(I will edit first post to mention there are more interesting results further down)

I you look at the median times of the Hammerhead vs weak target test, they look awfully close (52 vs 54 vs 56). One should keep in mind that the measured time is effectively: time to reach target + time to destroy target + additional time before the timer stops. If we were to subtract both the time to reach and additional time from the measurement, we would look at much lower numbers, maybe around 20 seconds. Also, since the tests are less noisy than my first attempt and since there were 24 repetitions, those measurements have a good enough quality IMO, both individually and as a data set. In any cases, looking at the distribution of time proved useful in order to check results and compare performance of guns, which brings us to…

Bonus: visualization of time distribution

(warning: shiny graphics sure are good, but only if you keep in mind what was tested)





Logged

Thaago

  • Global Moderator
  • Admiral
  • *****
  • Posts: 7174
  • Harpoon Affectionado
    • View Profile
Re: 0.9.1a weapon testing - Thumper experiment
« Reply #11 on: July 18, 2020, 11:31:52 AM »

Fantastic testing!

I am very surprised that 2 heavy mortars did so much better than flaks against fighters! I wonder if this is because of the flares on the gladius distract the PD tagged flaks? In either case, heavy mortars ftw!
Logged

Igncom1

  • Admiral
  • *****
  • Posts: 1496
    • View Profile
Re: 0.9.1a weapon testing - Thumper experiment
« Reply #12 on: July 18, 2020, 01:06:10 PM »

The age of the Arbalests is over. The age of the Heavy Mortar has begun!

Spoiler
[close]
Logged
Sunders are the best ship in the game.

RustyCabbage

  • Captain
  • ****
  • Posts: 347
    • View Profile
Re: 0.9.1a weapon testing - Thumper experiment
« Reply #13 on: July 18, 2020, 07:42:19 PM »

The updated tests look very solid! I suspect the flares are stopping the Flaks from performing their anti-fighter as well as they should, but great to see that the Thumper consistently underperforms against the Mortar.

I guess there might be a niche against frigates and some of the faster destroyers, where the Mortars will probably miss many more shots, but given how they are beaten even against fast, low armored fighters that might be wishful thinking.

pairedeciseaux

  • Captain
  • ****
  • Posts: 340
    • View Profile
Re: 0.9.1a weapon testing - Thumper experiment
« Reply #14 on: July 18, 2020, 11:13:45 PM »

I am very surprised that 2 heavy mortars did so much better than flaks against fighters! I wonder if this is because of the flares on the gladius distract the PD tagged flaks?

I suspect the flares are stopping the Flaks from performing their anti-fighter as well as they should,

You guys are right. The Flak did prioritize shooting at flares, did a good job at it, then was able to concentrate on Gladius. I should have mentioned it because it is indeed an unfair situation for Flak compared to other tested guns as far as shooting Gladius is concerned.

On the other hand this is not really a detrimental simulation for Flak as long as you consider the gun did provide both effective anti-missiles protection and measurable anti-fighter protection. If I ever do a serious PD gun test or serious fighters test, I should then try to measure these 2 things separately, if at all possible.

For instance, using Spark instead of Gladius would give different results, because no flare and less resilient. Using Thunder, which as same armor and hull as Gladius, would still bring other fun differences because of swarmer missiles and ion.
Logged