Fractal Softworks Forum

Starsector => Mods => Modding => Topic started by: NoFoodAfterMidnight on October 17, 2019, 07:20:29 AM

Title: Releasing a mod that overwrites some files in the .api
Post by: NoFoodAfterMidnight on October 17, 2019, 07:20:29 AM
I've been working on a difficulty mod that adds a lot of features and big changes to Starsector, and the only way I could do it without breaking compatibility with content mods (which is very important), or do it at all, was to overwrite some of the class files in the starfarer.api.JAR. Is it okay to upload a mod like that to this forum? It's compatible with every mod on the forum since no other mods do this, but since it overwrites some game files I thought I'd ask.
Title: Re: Releasing a mod that overwrites some files in the .api
Post by: Snrasha on October 17, 2019, 08:02:44 AM
I've been working on a difficulty mod that adds a lot of features and big changes to Starsector, and the only way I could do it without breaking compatibility with content mods (which is very important), or do it at all, was to overwrite some of the class files in the starfarer.api.JAR. Is it okay to upload a mod like that to this forum? It's compatible with every mod on the forum since no other mods do this, but since it overwrites some game files I thought I'd ask.

Many mods do that, but for the part of compatibility, you have really checked every top-tier popular mods?

Nexerelin overwrite many files of the game like the market and you need to override marketAPI of Nexerelin if you want have special market per exemple.
Release it on the Modding first and people will see if this is really compatible with many mods.
After
Quote
I've been working on a difficulty mod that adds a lot of features and big changes to Starsector,
Look like a "balance mod", we have already 5-6 mods like that ^^, so no issue to post it. If you want post it on the mod list, this thing will be probably a total conversion mod, i think, or Megamods.

Because yeah, some mods like that "break" every content mods which are based their balance around the vanilla. But do not fear and post it.

Edit: after have reread my message and your message, no, yeah, your mod is more a Nex-like?



Title: Re: Releasing a mod that overwrites some files in the .api
Post by: Alex on October 17, 2019, 09:05:48 AM
I've been working on a difficulty mod that adds a lot of features and big changes to Starsector, and the only way I could do it without breaking compatibility with content mods (which is very important), or do it at all, was to overwrite some of the class files in the starfarer.api.JAR. Is it okay to upload a mod like that to this forum? It's compatible with every mod on the forum since no other mods do this, but since it overwrites some game files I thought I'd ask.

Many mods do that, but for the part of compatibility, you have really checked every top-tier popular mods?

Nexerelin overwrite many files of the game like the market and you need to override marketAPI of Nexerelin if you want have special market per exemple.
Release it on the Modding first and people will see if this is really compatible with many mods.

I'm pretty sure Nexerlin does this without requiring the user to overwrite their starfarer.api.jar with another version of the file.

To answer the question, I have some reservations about a mod doing this. Not the least is that *right now* this would be the only mod doing it, but if this was an accepted way of doing things, that would change in a hurry, and we'd have a bunch of mutually-incompatible mods.

I'd much rather these kinds of things be done through the modding API, and, where that's not possible, result in requests for API changes/additions, since in that case, we'd be improving the API for everyone. That's slower going, certainly, but I think things would end up in a better place in the long run.
Title: Re: Releasing a mod that overwrites some files in the .api
Post by: NoFoodAfterMidnight on October 17, 2019, 09:51:32 AM
The mod is fully compatible with nexerelin and ruthless sector (I play with both), and every mod that adds content. It's built to be compatible with all the faction and content mods, and integrates Nexerelin for some of it's functions. It does things no other mod does currently, and all the changes are to core mechanics and don't require content mods to add anything to be compatible. If it's released and other mods start doing similar things with overwriting the same game files it would start to become incompatible with those mods, however.

To answer the question, I have some reservations about a mod doing this. Not the least is that *right now* this would be the only mod doing it, but if this was an accepted way of doing things, that would change in a hurry, and we'd have a bunch of mutually-incompatible mods.

I'd much rather these kinds of things be done through the modding API, and, where that's not possible, result in requests for API changes/additions, since in that case, we'd be improving the API for everyone. That's slower going, certainly, but I think things would end up in a better place in the long run.

Yeah I was afraid of setting a precedent, but there's already a few essentially incompatible mods due to their content/vision, and some that are just incompatible.

There's a lot of stuff I can't do within the API, like changing how FleetFactoryV3 works and generates fleets, and how ship salvaging works. If there's a way to overwrite or alter vanilla plugins through the API, then the total changes to the API would be FleetFactoryV3, RemnantSeededFleetManager, DmodManager, BuffmanagerAPI, LuddicPathCellsIntel, and ShipRecoverySpecial. There's about a dozen alterations to existing plugins.

It's a pretty big project and I'd love to see people in the community using it. I'd like to release the mod soon and then update it when the API gets updated, but if that's not okay with you I'll keep the mod between friends so it doesn't become a thing.
Title: Re: Releasing a mod that overwrites some files in the .api
Post by: Alex on October 17, 2019, 01:21:02 PM
There's a lot of stuff I can't do within the API, like changing how FleetFactoryV3 works and generates fleets, and how ship salvaging works. If there's a way to overwrite or alter vanilla plugins through the API, then the total changes to the API would be FleetFactoryV3, RemnantSeededFleetManager, DmodManager, BuffmanagerAPI, LuddicPathCellsIntel, and ShipRecoverySpecial. There's about a dozen alterations to existing plugins.

Made a note of these; some are more straightforward to expose via plugins than others.

As far as RemnantSeededFleetManager and LuddicPathCellsIntel, as well as ShipRecoverySpecial - it *seems* like it should be possible to override these "normally"; am I missing some aspect of this?

More detail about what you need from the others would be good as well so that I can make them pluggable in a sensible way.

It's a pretty big project and I'd love to see people in the community using it. I'd like to release the mod soon and then update it when the API gets updated, but if that's not okay with you I'll keep the mod between friends so it doesn't become a thing.

Yeah, I'd hate to say "no" given that you've already put a bunch of work into it, and, frankly, it sounds interesting. So I'd say, at least provisionally, go for it!

If anyone from the modding community has thoughts/wants to chime in on this, that's definitely welcome.
Title: Re: Releasing a mod that overwrites some files in the .api
Post by: Sundog on October 17, 2019, 05:30:13 PM
I've been working on a difficulty mod that adds a lot of features and big changes to Starsector
I'm looking forward to finding out how this turns out! Hopefully this means you've already implemented some of my plans for ruthless sector so I can cross a few things off my list  ;D
Title: Re: Releasing a mod that overwrites some files in the .api
Post by: MesoTroniK on October 18, 2019, 01:56:42 AM
Since we cannot really anticipate any possible issues without it, can you please post a list of replaced scripts and also maybe broadly what you did overall? I would like to believe you, but to be perfectly honest, I can't help but to suspect that what you been working on will affect some current or future mods.
Title: Re: Releasing a mod that overwrites some files in the .api
Post by: NoFoodAfterMidnight on October 18, 2019, 12:25:40 PM
As far as RemnantSeededFleetManager and LuddicPathCellsIntel, as well as ShipRecoverySpecial - it *seems* like it should be possible to override these "normally"; am I missing some aspect of this?

I think you're right, they're extensions of extensions of plugins, I was just going through my list of changed files and checking for "plugin". I'm not quite sure how to overwrite or extend them through the plugin system.

I extended the buff interface of BuffManagerAPI to include an option for retrieving the "level" and "name" of the buffs.

In DModManager I changed addDMods() to change the way Dmods were added, how many and which ones it chooses to use

In FleetFactoryV3 I changed the CreateFleet() and addCommanderAndOfficers(), altering and inserting a bunch of things that needed to happen during the creation of every fleet. This could actually be solved just by having a fleet creation listener and altering the fleet post creation through an inflater and some scripts I think.

I also changed BaseIndustry to support more installable items and cores

Yeah, I'd hate to say "no" given that you've already put a bunch of work into it, and, frankly, it sounds interesting. So I'd say, at least provisionally, go for it!

Cool :)

Since we cannot really anticipate any possible issues without it, can you please post a list of replaced scripts and also maybe broadly what you did overall? I would like to believe you, but to be perfectly honest, I can't help but to suspect that what you been working on will affect some current or future mods.

This mod is essentially a difficulty overhaul with big features to extend the end game and make the game more challenging past a certain point. It isn't going to be compatible with every balance changing mod or tweak, but is designed to work with all content adding mods and major feature adding mods that don't change vanilla balance.

Unless something also alters the same files in the .api in the future, there shouldn't be any conflicts even in future content or feature mods, except maybe with anything that replaces vanilla plugins like FleetEncounterContext to add features. Nexerelin and Ruthless Sector both do that and are compatible though.

There are some *balance* conflicts with some mods, like Gladiator or anything that makes huge fleets, I can't say for certain whether or not those will be balanced at all, but they will work. And probably not compatible with any other broad balance changing mods that alters vanilla content, and possibly some tweak mods, just out of differences in direction.

In broad strokes: the biggest change is the alteration of FleetFactoryV3 and the addition of fleet level buffs. Fleets get assigned a buff that affects all their ships' combat stats when the fleets get more FP. The fleet sizes get diminishing returns so instead of 30 capital ships in an enemy fleet, there's maybe 7 + cruisers and below, but they're high level. This is all done as fleets are created through FleetFactoryV3, which everything in the game uses to generate fleets. It also changes officer assignment and gives officers specialty buffs, which is also all done through FleetFactoryV3. Fleet sizes in general are also larger depending on a multitude of different factors, all done through FleetFactoryV3 again. FleetParams is unchanged, it requires nothing from any script that makes fleets not needed in vanilla, so every mod that uses a script to generate a fleet is compatible.

Full list:

.API files changed:

FleetFactoryV3
DModManager (might have balance issues with anything that adds Dmods due to changes in tags used)
BuffManagerAPI (Anything that adds out of combat buffs may not be compatible, unsure)
BaseIndustry (new industries inherit the changes/additions, so no conflict with industry mods like terraforming)

Plugins Changed:

FleetEncounterContext
DerelictShipEntityPlugin
AICores
SalvageDefenderInteraction
RemnantSeededFleetManager
RemnantStationFleetManager
RemanntThemeGenerator
ShipRecoverySpecial
LuddicPathCellsIntel

Plugins and Scripts added:

A bunch of buff scripts
A MagicUI derivative for in-combat buff display
An EveryCombatFrame plugin

Vanilla Content Rebalanced through normal means:

Most skills
Most Industries
All Commodities
Most Hullmods
Salvage_Entity_Gen_Data.csv
Drop_Groups.csv
Settings.json

Vanilla ship and weapon balance is unchanged, so any faction or content mod will be thematically compatible with the overhaul.





Title: Re: Releasing a mod that overwrites some files in the .api
Post by: Sundog on October 18, 2019, 04:17:11 PM
Unless something also alters the same files in the .api in the future, there shouldn't be any conflicts even in future content or feature mods, except maybe with anything that replaces vanilla plugins like FleetEncounterContext to add features. Nexerelin and Ruthless Sector both do that and are compatible though.
FleetEncounterContext is already contested real estate. In addition to Nex and ruthless sector, SWP overwrites it for IBB bounties and MesoTroniK's upcoming THI update will overwrite it as well. The only reason ruthless sector doesn't conflict with those other mods is that I wrote a FleetEncounterContext specifically for each one of them that provides the functionality of both ruthless sector and the other mod. If your mod overwrites FleetEncounterContextImpl directly in starfarer.api.jar, then I suspect all five of these mods will overwrite your own since they normally overwrite vanilla.
Title: Re: Releasing a mod that overwrites some files in the .api
Post by: Thaago on October 18, 2019, 05:15:55 PM
Oof... given 5 (!) mods overwrite that one, what API changes do you think would be needed to remove that?
Title: Re: Releasing a mod that overwrites some files in the .api
Post by: NoFoodAfterMidnight on October 18, 2019, 07:10:58 PM
I'm changing getRecoverableShips() in FleetEncounterContext, which doesn't get modified by RS, Nex, or SWP currently.
Title: Re: Releasing a mod that overwrites some files in the .api
Post by: Alex on October 18, 2019, 07:17:29 PM
... and those mods then derive from it, so if they don't change getRecoverableShips(), they'll just inherit the functionality and it would work. At least, that's what I'm getting from this.
Title: Re: Releasing a mod that overwrites some files in the .api
Post by: Sundog on October 18, 2019, 10:13:27 PM
Oof... given 5 (!) mods overwrite that one, what API changes do you think would be needed to remove that?
Well, the API could maybe provide listeners for some of the functionality people get from overriding FleetEncounterContexts, but that would have complications, most of which I probably wouldn't anticipate. Alex might have some bright ideas, but I can't come up with anything better than "if it ain't broke don't fix it."

I'm changing getRecoverableShips() in FleetEncounterContext, which doesn't get modified by RS, Nex, or SWP currently.
... and those mods then derive from it, so if they don't change getRecoverableShips(), they'll just inherit the functionality and it would work. At least, that's what I'm getting from this.
Oh, that's right. I was thinking custom FleetEncounterContexts would continue to extend the vanilla Impl version, but obviously that wouldn't be the case if it were replaced in starfarer.api.jar.

I just checked, and SWP actually does override getRecoverableShips. However, it calls the super-method, so there wouldn't be any conflicts anyway.

Title: Re: Releasing a mod that overwrites some files in the .api
Post by: Nicke535 on October 19, 2019, 02:00:44 AM
While I'm not gonna go all "Don't do it, you madman!", I am going to say that overwriting the API pretty much ruins the point of working towards an API in the first place.

While some of these changed files may very well have justified reasons for being changed, it's as many has pointed out before: changing the API is very dangerous territory. Because it breaks the "promise" provided by the default API, no future mods can make assumptions on how things operate; aforementioned changes to the API from existing mods are already skimming the line and, as mentioned, can already cause compatibility issues. With these changes, it's pretty much impossible for anyone to know which mods are compatible in which situations without explicitly digging through all source code of all mods affected and comparing them towards the now-changed API.
Title: Re: Releasing a mod that overwrites some files in the .api
Post by: Sundog on October 19, 2019, 02:18:56 PM
@Nicke535: Fundamentally, I absolutely agree. Changing the API means altering the tools all mod authors use, meaning we could potentially end up needing to check which version of the API we're using to ensure appropriate functionality. On the other hand, it doesn't take overriding the API to introduce that sort of problem. Any mod that adds or alters features has the potential to introduce complications for other mods. For example, faction mods often need to change how they behave based on whether or not Nex is active. If it's handled well, NoFoodAfterMidnight's mod could end up requiring very little consideration on the part of other mods in spite of changing the API. I think what is changed is more important than how it's changed.

@NoFoodAfterMidnight: Another potential issue is that any exception that occurs within your code will look like it took place in vanilla code. That could make things very difficult to troubleshoot, especially for anyone who doesn't know about this aspect of your mod's implementation. It might be worth catching and re-throwing any error that occurs in your overriden API methods so you can add a message to the log about how your mod encountered an error.
Title: Re: Releasing a mod that overwrites some files in the .api
Post by: Alex on October 19, 2019, 09:01:25 PM
@NoFoodAfterMidnight: Another potential issue is that any exception that occurs within your code will look like it took place in vanilla code. That could make things very difficult to troubleshoot, especially for anyone who doesn't know about this aspect of your mod's implementation. It might be worth catching and re-throwing any error that occurs in your overriden API methods so you can add a message to the log about how your mod encountered an error.

Oh, that's a super good point, and, uh, that could really blindside me. Glad you brought it up, since I'm at least directly aware of it now.
Title: Re: Releasing a mod that overwrites some files in the .api
Post by: NoFoodAfterMidnight on October 20, 2019, 11:45:46 AM
@NoFoodAfterMidnight: Another potential issue is that any exception that occurs within your code will look like it took place in vanilla code. That could make things very difficult to troubleshoot, especially for anyone who doesn't know about this aspect of your mod's implementation. It might be worth catching and re-throwing any error that occurs in your overriden API methods so you can add a message to the log about how your mod encountered an error.

That's a great point, I didn't think about that. I've taught myself java up until now, exceptions/error reporting in any language is pretty complex, so to make sure I do it right and not butcher the error/stack trace, how would you guys do it?
Title: Re: Releasing a mod that overwrites some files in the .api
Post by: Sundog on October 20, 2019, 05:47:39 PM
Well, there's likely a more elegant solution, but what I had in mind was something like this:
Code: java
public class FleetFactoryV3 {
public static CampaignFleetAPI createFleet(FleetParamsV3 params) {
try {
myObject.doModStuff();

vanillaObject.doVanillaStuff();
} catch (Exception e) {
Global.getLogger(FleetFactoryV3.class).error("NoFoodAfterMidnight's difficulty mod encountered an error!");

throw e;
}
}
}
This would ensure that the game crashes and logs the stack trace as normal, only preceded by a message identifying you as the appropriate point of contact and leaving a clue about the stack trace referring to your modified API instead of the vanilla one.
This approach would fail if you were to modify a field (member variable) in such a way that it caused an exception somewhere that isn't wrapped in a try block, so I would be extra careful to avoid that.
Title: Re: Releasing a mod that overwrites some files in the .api
Post by: Nick XR on October 27, 2019, 06:30:10 PM
Another concern would be security.  I'm pretty sure janino is running in a Security Manager constrained manner, but if an API altering mod could include some hostile code that could do all sorts of naughty things if the user is running in Admin mode (which probably a small % of people are).  All it would take is for someone to take over a popular defunct mod and inject that, or to make their own mod and include bad bits.

Something like this:  https://thehackernews.com/2018/11/nodejs-event-stream-module.html
Although NPM is a dumpster fire when it comes to security so finding an exploit related to NPM is easy.
Title: Re: Releasing a mod that overwrites some files in the .api
Post by: Alex on January 19, 2021, 12:15:30 PM
Bit of a necro, but: added plugins that allow replacing FleetFactoryV3.createFleet() and DModManager's various .addDMods() methods.