(posted on 2020-07-13, then edited on 2020-07-18 to adjust the TL;DR and fix a typo)So I wanted to evaluate the usefullness of the Thumper medium ballistic weapon, mainly to learn about the testing methodology and tools, less so for the insight about Thumper.
I suspected it would be hard to gain knowledge on Thumper using repetitive tests while keeping a realistic simulation, but still wanted to try. What I mean by realistic is I don't want to perform micro-benchmarks measuring just the effect of the weapon on a dumb target, and ideally I want to recreate conditions that exists in real campaigns. But whether a success or failure, it would serve as a good preperation for more tests.
TL;DR - Overall replacing 1 Flak by 1 Thumper on Enforcer did not bring benefit in these tests, either because it really does not, or because the test failed to measure it.
Update: You can safely stop reading here unless you are interested in the testing process or subtle fleet battle issues. The first experiment was really flawed, I got better measurements in my second attempt, details in another message below. Short version is: Heavy Mortar has better performance than Thumper in these tests.Testing effort inspired by:
https://fractalsoftworks.com/forum/index.php?topic=18804and:
https://fractalsoftworks.com/forum/index.php?topic=15758DisclaimerThis is my first attempt at repetitive Starsector fleet testing and I'm no pro statistician. Don't put too much faith in data or statistics shown below.
Ships and loadoutI used Enforcers as a test platform, with a custom loadout, from left to right: 1 Flak, 1 Arbalest, 1 Heavy Mortar, 1 Arbalest, 1 Flak. And also 4x2 Hammers in front, Reinforced Bulkhead, Flux Distributor and Auxiliary Thrusters. This is the reference variant. I also tested a variant where I swapped the right-side Flak for a Thumper.
On the other side, I used Condors, the Attack variant: 2 Broadsword wings, 1 Pillum, 2 Vulcan. This is the reference variant. I also used one variant without any fighter, and another variant without any missile.
All fights had 2 Enforcers against 2 Condors, all with aggressive personnality.
MethodI mesured the time to destroy Condors accross all matchups, and did 18 repetitions of each. I didn't use results when Condor retreated. Statistic analysis gave a few data points, including median time and time distribution accoss 1 minute wide segments (number of samples per segment).
Starsector 0.9.1a-RC8, by FractalSoftworks
Fleet Tester mod 1.0, by Tartiflette
MagicLib 0.28, by Tartiflette
LazyLib 2.4e or f, by LazyWizard
Running in 1024x768 in windowed mode, with sound disabled.
I ran 3 simultaneous battles in 3 Starsector instances, each running in real time (no speed increase).
So 3 windows in the top area of the screen, with a slight overlap, looking at the 3 battle map.
And 3 terminal windows, manually starting/stopping "/usr/bin/time sleep 600".
Manually writing times in a speadsheet. Speadsheet also used to perform statistic analysis.
ResultsAgainst 2 Condors with 2 Broadswords and 1 Pillum
2 Enforcers with 2 Flaks are slightly faster than 2 Enforcers with 1 Flak and 1 Thumper
the median time difference is around 7%, 191 vs 204
time distribution:
2 to 3 minutes: 8 vs 7
3 to 4 minutes: 7 vs 7
4 to 5 minutes: 1 vs 2
over 5 minutes: 2 vs 2
Against 2 Condors with 1 Pillum (no fighters)
2 Enforcers with 2 Flaks are slightly faster than 2 Enforcers with 1 Flak and 1 Thumper
the median time difference is around 12%, 98 vs 110
time distribution:
1 to 2 minutes: 16 vs 15
2 to 3 minutes: 2 vs 3
Against 2 Condors with 2 Broadswords (no Pillum)
2 Enforcers with 2 Flaks are slightly slower than 2 Enforcers with 1 Flak and 1 Thumper
the median time difference is around 8%, 227 vs 209
time distribution:
2 to 3 minutes: 4 vs 4
3 to 4 minutes: 6 vs 9
4 to 5 minutes: 6 vs 3
over 5 minutes: 2 vs 2
An additial comparison worth looking at, with 2 Enforcers with 2 Flaks
2 Condors with 2 Broadswords and 1 Pillum is faster (at being destroyed) than 2 Condors with 2 Broadswords (no Pillum)
the median time difference is around 16%, 191 vs 227
2 to 3 minutes: 8 vs 4
3 to 4 minutes: 7 vs 6
4 to 5 minutes: 1 vs 6
over 5 minutes: 2 vs 2
CommentsI'm not 100% convinced about the sample size. I got 12 initialy, then moved to 18: it did not change the time distribution very much, it did not change the faster/slower situation. So it looks good, but I'm not sure. ;-)
I wrote above that I measured the time to destroy Condors. What happened in the simulation was more like:
time spent to travel to Condors (offense)
time spent to deal with fighters (defense)
time spent to deal with missiles (defense)
time spent to vent (offense and defense)
time spent to deal actual damage to Condors (offense)
I don't have the breakdown, but we can see the no fighters tests were twice as fast as the tests with fighters.
So what I tried to measure was only a small part of the actual measurement. Signal to noise ratio not that good. Not really a surprise, but a better test design could help.
In the end those tests showed Enforcer behaviour variation depending on external pressure. In other words, I tested defense againts this pressure as much as offense against the target ships. Which is not a bad thing to do with respect to the Thumper: this weapon should provide value against both fighters and ships. But doing it in the same test is hard IMO. Testing defense could be done separately - such as measuring time to loose against a swarm of fighters, carrier being impossible to catch.
Speaking about offense and defense, Enforcers shot Hammers a lot against both Broadswords and Condors. IMO, this reduced my ability to measure differences between 2 Flaks and 1 Flak + 1 Thumper. But I didn't want to run a test with a missile-less Enforcer because it is not realistic / not my preference (well, it's a player choice, so it could be), so I'm still confortable with that choice. Still... it might be worth checking in another test setup.
The Broadswords sure did put a lot of pressure on the Enforcers. The temporary stunlock effect is quite impressive. It is somewhat representative of what a fighter heavy fleet would impose on its opponent (Legion with 4 Broadsword wings would do this). I added the no fighters matchups to check results without this pressure. Interesting to see the result difference ... and similitude.
The Pillum situation is less clear.
With Pillum pressure I got shorter time than without Pillum pressure. I've got 2 mad theories on this. (1) Enforcer with dual Flaks got improved averall DPS while damaging missiles and fighters simultaneouly. Hmm, that does not sound right. (2) Condors without Pillum where less compelled to stay in sensor range, so could kite from a longer distance. Hmmm (3) or results are just plain wrong.
Tentative conclusionDid 1 Thumper on Enforcer provide measurable benefits in the no fighter tests? No it was worse according to statistics.
Did 1 Thumper on Enforcer provide measurable benefits in the tests with Broadswords? Yes in 1 case (no Pillum) and no (it was worse) in 1 case (Pillum), according to statistics. So no expected overall benefits considering mixed real campaign situations.
If it is so difficult with Enforcer, my brain hurts trying to imagine testing effect of having 1 or 2 Thumpers on Onslaugth or Legion with realistic loadout (both being candidate because they have plenty medium ballistic turrets).
Comments and questions welcome! Also having more than one pair of eyes checking results and methodology sure is usefull.
Dataplayer0_fleet.csv
Spoiler
rowNumber,variant,personality,flagship
1,enforcer_Custom_FlakAndThumper,aggressive,false
2,enforcer_Custom_FlakAndThumper,aggressive,false
player1_fleet.csv
Spoiler
rowNumber,variant,personality,flagship
1,condor_Attack_Custom_NoPillum,aggressive,false
2,condor_Attack_Custom_NoPillum,aggressive,false
enforcer_Custom_FlakOnly.variant
Spoiler
{
"displayName": "Custom FlakOnly",
"fluxCapacitors": 15,
"fluxVents": 20,
"goalVariant": true,
"hullId": "enforcer",
"hullMods": [
"reinforcedhull",
"fluxdistributor",
"auxiliarythrusters"
],
"permaMods": [],
"quality": 0.5,
"variantId": "enforcer_Custom_FlakOnly",
"weaponGroups": [
{
"autofire": false,
"mode": "ALTERNATING",
"weapons": {
"WS 006": "hammer",
"WS 007": "hammer",
"WS 008": "hammer",
"WS 009": "hammer"
}
},
{
"autofire": true,
"mode": "LINKED",
"weapons": {"WS 003": "heavymortar"}
},
{
"autofire": true,
"mode": "LINKED",
"weapons": {
"WS 001": "flak",
"WS 005": "flak"
}
},
{
"autofire": true,
"mode": "LINKED",
"weapons": {
"WS 002": "arbalest",
"WS 004": "arbalest"
}
}
]
}
enforcer_Custom_FlakAndThumper.variant
Spoiler
{
"displayName": "Custom FlakAndThumper",
"fluxCapacitors": 14,
"fluxVents": 20,
"goalVariant": true,
"hullId": "enforcer",
"hullMods": [
"reinforcedhull",
"fluxdistributor",
"auxiliarythrusters"
],
"permaMods": [],
"quality": 0.5,
"variantId": "enforcer_Custom_FlakAndThumper",
"weaponGroups": [
{
"autofire": false,
"mode": "ALTERNATING",
"weapons": {
"WS 006": "hammer",
"WS 007": "hammer",
"WS 008": "hammer",
"WS 009": "hammer"
}
},
{
"autofire": true,
"mode": "LINKED",
"weapons": {"WS 003": "heavymortar"}
},
{
"autofire": true,
"mode": "LINKED",
"weapons": {
"WS 001": "flak"
}
},
{
"autofire": true,
"mode": "LINKED",
"weapons": {
"WS 005": "shredder"
}
},
{
"autofire": true,
"mode": "LINKED",
"weapons": {
"WS 002": "arbalest",
"WS 004": "arbalest"
}
}
]
}
condor_Attack_Custom_NoFighter.variant
Spoiler
{
"displayName": "CustomNoFighters",
"fluxCapacitors": 7,
"fluxVents": 7,
"goalVariant": true,
"hullId": "condor",
"hullMods": [],
"permaMods": [],
"variantId": "condor_Attack_Custom_NoFighter",
"weaponGroups": [
{
"autofire": false,
"mode": "ALTERNATING",
"weapons": {"WS 003": "pilum"}
},
{
"autofire": true,
"mode": "LINKED",
"weapons": {
"WS 001": "vulcan",
"WS 002": "vulcan"
}
}
],
"wings": [
]
}
condor_Attack_Custom_NoPillum.variant
Spoiler
{
"displayName": "CustomNoPillum",
"fluxCapacitors": 7,
"fluxVents": 7,
"goalVariant": true,
"hullId": "condor",
"hullMods": [],
"permaMods": [],
"variantId": "condor_Attack_Custom_NoPillum",
"weaponGroups": [
{
"autofire": true,
"mode": "LINKED",
"weapons": {
"WS 001": "vulcan",
"WS 002": "vulcan"
}
}
],
"wings": [
"broadsword_wing",
"broadsword_wing"
]
}
Matchups CSV
Spoiler
;2 Enforcers with 2 Flaks;2 Enforcers with 1 Flak and 1 Thumper
2 Condors with 2 Broadswords and 1 Pillum;set 1A;set 1B
2 Condors with 1 Pillum (no fighters);set 2A;set 2B
2 Condors with 2 Broadswords (no Pillum);set 3A;set 3B
Raw data CSV
Spoiler
;set 1A;set 1B;empty;set 2A;set 2B;empty;set 3A;set 3B
1;225;241;;99;99;;226;200
2;162;177;;97;119;;170;212
3;390;212;;120;130;;228;320
4;197;196;;96;127;;259;241
5;223;215;;85;113;;211;195
6;166;306;;94;110;;197;130
7;206;131;;113;101;;288;213
8;135;320;;100;97;;244;261
9;215;175;;105;89;;167;151
10;136;221;;89;167;;263;190
11;314;269;;83;110;;347;208
12;172;157;;91;112;;164;179
13;136;150;;130;103;;376;303
14;171;185;;89;101;;219;209
15;185;142;;140;111;;192;289
16;262;231;;117;94;;151;216
17;214;227;;84;77;;283;180
18;133;156;;105;120;;264;199
Statistical analysis CSV
Spoiler
;set 1A;set 1B;empty;set 2A;set 2B;empty;set 3A;set 3B;interval values;interval indexes
min;133;131;;83;77;;151;130;;
max;390;320;;140;167;;376;320;;
average;202;206;;102;110;;236;216;;
median;191;204;;98;110;;227;209;;
max - min;257;189;;57;90;;225;190;;
stddev;67;54;;16;19;;62;50;;
avedev;48;43;;13;13;;48;37;;
freq 0 to 1 min;0;0;;0;0;;0;0;60;1
freq 1 to 2 min;0;0;;16;15;;0;0;120;2
freq 2 to 3 min;8;7;;2;3;;4;4;180;3
freq 3 to 4 min;7;7;;0;0;;6;9;240;4
freq 4 to 5 min;1;2;;0;0;;6;3;300;5
freq over 5 min;2;2;;0;0;;2;2;;6