Fractal Softworks Forum

Please login or register.

Login with username, password and session length
Advanced search  

News:

Starsector 0.97a is out! (02/02/24); New blog post: Planet Search Overhaul (07/13/24)

Pages: 1 [2] 3 4

Author Topic: Starsector Repo - Starsector Modding Repository  (Read 9549 times)

Uomoz

  • Admiral
  • *****
  • Posts: 2663
  • 'womo'dz
    • View Profile
Re: Starsector Repo - Starsector Modding Repository
« Reply #15 on: January 14, 2020, 03:58:35 AM »

That would probably work but would also have a bunch of possible problems related to the automated process (a lot of elements are missing or completely custom depending on the post itself, like changelog or compatibility). Also it would probably look pretty ugly, and would miss a lot of functionalities I'd love to add like tags, batch download, compatibility etc.

That said the idea of scraping the mod forum is pretty interesting as a seeder for the repo.
The Repo could then be automated like that but also accept the modder intervention (after the ownership is verified) so the post it could be properly customized with a premade starting point.
Logged

Uomoz

  • Admiral
  • *****
  • Posts: 2663
  • 'womo'dz
    • View Profile
Re: Starsector Repo - Starsector Modding Repository
« Reply #16 on: January 14, 2020, 07:28:26 AM »

Sadly retrieving dl links from the topic seems very hard, simply because there are usually tons of them in the mod first post, and are usually not directly part of a link tag with the Download word in it...



EDIT:

Interestingly enough the whole forum post is easily movable to the Repo as a temporary solution and since it can be crawled on request it will always be updated..

« Last Edit: January 14, 2020, 08:05:34 AM by Uomoz »
Logged

silentstormpt

  • Admiral
  • *****
  • Posts: 1060
    • View Profile
Re: Starsector Repo - Starsector Modding Repository
« Reply #17 on: January 14, 2020, 09:04:31 AM »

Nexus API allows you to do it and provides you with alot of info like changelog, url for their download on nexus, etc, only need to provide the mods id from there and get an API Key:

https://app.swaggerhub.com/apis-docs/NexusMods/nexus-mods_public_api_params_in_form_data/1.0
Logged

Uomoz

  • Admiral
  • *****
  • Posts: 2663
  • 'womo'dz
    • View Profile
Re: Starsector Repo - Starsector Modding Repository
« Reply #18 on: January 14, 2020, 09:35:31 AM »

That's very useful, to fill info for the mods that are actually present on the Nexus. That said there's no way to programmatically fetch those info without the id, using only the info i can fetch from the mod forum. Mmh.. I could set up a way so the Modder can one-click import all the relevant info in the website from the nexus db just using the api though..

Thanks for the help/feedback :)
Logged

Nick XR

  • Admiral
  • *****
  • Posts: 719
    • View Profile
Re: Starsector Repo - Starsector Modding Repository
« Reply #19 on: January 14, 2020, 03:00:22 PM »

Instead of having to have modders create a login or something, I think the only way you'll get adoption of this is the modders don't have to do anymore work than creating a JSON file that describes stuff like:
  • Name
  • Description
  • Title Graphic
  • Other Graphics


Maybe this lives in a robots.json in the produced artifact, or in a github/bitbucket (so it can work with closed or open source mods). 

To get this you just scrape discord/the forums downloading all zip files and scraping all github/bitbucket pointers to look for them.  Track things like zip file size to watch for changes.

Then put this data in some sort of DBish thing and build a UI to your hearts content.



I know the scraping is a bit of a PITA, but I don't think you're going to get modders to manage yet another mod repo.

Uomoz

  • Admiral
  • *****
  • Posts: 2663
  • 'womo'dz
    • View Profile
Re: Starsector Repo - Starsector Modding Repository
« Reply #20 on: January 14, 2020, 03:47:51 PM »

I'd say that the only true mod repo is Nexus, but I also believe it doesn't really work well with SS, the way is made for multiple games. Feels pretty generic to me.
Anyway, I'll make it so it's so effortless that eventually some modders will try it. The dream is to have modders and players actually use the thing over the other systems (the forum is NOT a good place to store Archives of things IMO, same for Discord and the Nexus can't possibly feature the nuances of the SS community/mod scene) but at the point I'm at already I have already populated the Repo with all the mods in the mod Index. The only thing missing is the DL button linking to the actual DL location, it's a bit of work on procedural url-guessing.
I found ways to guess changelogs pretty decently as well so that's out of the way.

Mass scraping all the posts/discord/possible-locations is out of the way, all mods have multiple files linked everywhere and often are older versions/irrelevant stuff, so it would be REALLY inefficient.

As it is right now it clones the form post in its entirety (with some css magic to make it a bit more pleasing) and it just works.

I was thinking about a "crowd sourced" tagging system where anons could add and remove tags based on general approval and sliders to vote on stuff like "crazy<--->lorefriendly", so it could be easy to find mods similar to your tastes.

Plenty ideas, thanks for all the feedback!
« Last Edit: January 14, 2020, 03:52:13 PM by Uomoz »
Logged

Wispborne

  • Captain
  • ****
  • Posts: 436
  • Discord: wispborne
    • View Profile
Re: Starsector Repo - Starsector Modding Repository
« Reply #21 on: January 17, 2020, 11:44:24 AM »

<pre-post edit> I wrote all of the below before I realized how senior a member you are (I spend the majority of my time on the modding section of the Discord channel). Please disregard anything that comes across as patronizing; I had incorrectly assumed that you were new to the forum and going cowboy without fully realizing how the scene worked, but you've been here wayyyyy longer than I</edit>


Nexus Mods has tags, categories, and mod dependencies already. It also has a one-click install mod manager, which prevents users from doing things like merging a new version of a mod into the old version. I think mod packs can be created by making a mod that contains custom/compatibility settings, then marking the "bundled" mods as dependencies.
The site is also established, meaning that it's unlikely to disappear if the maintainer loses interest or has life generally get in the way (bus factor), which is a fairly common thing and just a fact of life.

The main reason that Nexus Mods hasn't taken off for Starsector, I think, is that it hasn't taken off. Most mods are already in one place and authors don't want to spend time updating multiple places if they don't have to whenever they release an update.

Don't get me wrong, I'd love to see this succeed and you have my permission to add Gates Awakened, but there's a lot of inertia working against it. When I release a new version, I create a Github release (which I scripted using their API to take less time), then (manually) update my release on Discord, then (manually) update my forum page, then (manually) update Nexus Mods, and finally (manually) update version checker so that users receive an update notification. It's an annoying process already.

If info was scraped, that's fine. If I need to add some compatibility info to a json file, that's fine for me, but I know that many prevalent modders treat save-compatibility as fairly unimportant and sometimes leave it up to users to report whether an update is compatible or not. This is a hobby, not a profession, so it's unfair and unreasonable to expect modders to prioritize save-compatibility (although it should be encouraged).

Sorry to be a downer. I tend to focus on making cool and fun things before thinking about whether they're the right things to make, so I totally get where you're coming from, and really do hope it works out.
Logged
Mod: Persean Chronicles | Mod Managers: SMOL & TriOS | Tool: VRAM Estimator | Tool: Forum+Discord Mod Database | If I'm inactive for 3 months, anyone can use any of my work for anything (except selling it or its derivatives).

Uomoz

  • Admiral
  • *****
  • Posts: 2663
  • 'womo'dz
    • View Profile
Re: Starsector Repo - Starsector Modding Repository
« Reply #22 on: March 04, 2020, 04:10:32 AM »

It's working, mostly.

The thing parses the forum for links from a bunch of different sources (direct link, git, drive etc). Opens the archives, parses mod_json or .version files and try to guess the most accurate data. Automatically update the mod record when the forum post changes.

It's really cool! And it's completely automated. All the information stored in this are publicly available and exactly as found around the forum.

What do you guys think about it?
I'd also love the general opinion about sharing this into the interwebs.. Should every modder be asked for permission for the mirroring of the mod forum post?



Logged

Nick XR

  • Admiral
  • *****
  • Posts: 719
    • View Profile
Re: Starsector Repo - Starsector Modding Repository
« Reply #23 on: March 04, 2020, 09:17:14 AM »

Looks amazing!  Any thoughts on categorization / tags?  My instinct is you should request that modders add some JSON file that contains meta-data that you look for.  Even if it's rather spartan now, you'll probably want it eventually.

Sundog

  • Admiral
  • *****
  • Posts: 1736
    • View Profile
Re: Starsector Repo - Starsector Modding Repository
« Reply #24 on: March 04, 2020, 05:11:01 PM »

Hey Uomoz, glad to see you back! This is a cool idea. I had some serious doubts at first, but scraping and displaying forum pages makes this seem a lot more feasible to me.

I can only speak for myself, but it wouldn't bother me to have my mods scraped into your repo without my explicit permission.

Looking forward to seeing where this goes, especially with regard to any features you implement to set it apart from the Nexus!

Harmful Mechanic

  • Admiral
  • *****
  • Posts: 1340
  • On break.
    • View Profile
Re: Starsector Repo - Starsector Modding Repository
« Reply #25 on: March 04, 2020, 05:26:55 PM »

Yeah, my only real concern is having to maintain yet another thing, so I'm also fine with it. Curious to see where it goes and how it develops.
Logged

AxleMC131

  • Admiral
  • *****
  • Posts: 1722
  • Amateur World-Builder
    • View Profile
Re: Starsector Repo - Starsector Modding Repository
« Reply #26 on: March 04, 2020, 09:00:37 PM »

On a whim I would suggest that you let mod authors have the opportunity to "blacklist" certain mods they don't want displayed for good reason. I notice in your example there that The Silent Armada is shown, despite being officially discontinued, and I'd prefer it if that doesn't get advertised for just that reason.

Other than that, having the whole system be automated is very smart.
Logged

Uomoz

  • Admiral
  • *****
  • Posts: 2663
  • 'womo'dz
    • View Profile
Re: Starsector Repo - Starsector Modding Repository
« Reply #27 on: March 05, 2020, 05:53:38 AM »

What about some sort of flag in either the mod_json or the .version file?

There's also another way around this, and it's to hide/display some data in the the forum post so that my cralwler knows some metadata like the official link to the compressed archive, or other flags.

What would you guys prefer, to add some stuff to the mod post or to do edit the mod files (either existing files or a new file)?

For example i'd love a way so the modder can specify the placement of the mod configuration file so a player could configure the mod even before downloading it :D
Logged

Nick XR

  • Admiral
  • *****
  • Posts: 719
    • View Profile
Re: Starsector Repo - Starsector Modding Repository
« Reply #28 on: March 05, 2020, 09:12:59 AM »

How about JSON embedded in a spoiler tag in the forum post?  That way you don't have to download every zip/7z/rar and pull it apart and guess which one is latest.

As for the JSON content, just off the top of my head:
  • Img URL
  • Download URL
  • compatible version (if it can't be figured out from the post title)
  • Mod Version (again if it can't be determined)

AxleMC131

  • Admiral
  • *****
  • Posts: 1722
  • Amateur World-Builder
    • View Profile
Re: Starsector Repo - Starsector Modding Repository
« Reply #29 on: March 05, 2020, 08:37:20 PM »

I'd be fine with something in the .version file since that makes sense, but not every mod supports version checker - and in fact I'm fairly sure TSA didn't. The mod_info.json file is probably the sanest place to put something like a "hey this mod is discontinued" tag, I don't mind updating the download link with such a change.

That said, Nick's got a point, if you can do it direct from the forum post, not only does it save the system having to download and look through the mod files, it also saves hassle for the author not having to do the above "update" to their mod. If it's possible to do that, it would be preferable.
Logged
Pages: 1 [2] 3 4