First off, thanks for the kind words about the AI
To answer your question... well, pretty much all code is a state machine in some sense, right? But for any more reasonable definition, the ship AI is pretty much a huge collection of if statements and special cases.
There are several independent modules - controlling movement, weapons fire, shields, flux venting, ship systems, etc. These communicate with each other by setting flags in a shared object. For example, one module might say "do not back off", "hold fire", "do not generate flux", etc, and other modules are coded to generally respect that, unless they have an overwhelming reason not to. There's also a 2nd-tier module whose job it is to look at longer-term trends and force behaviors based on that if necessary. An example of that might be detecting that the ship has been trying to back off for a while, but is not having success doing so.
You might note that this setup doesn't lend itself well to "planning" - i.e. there's nothing that would make the AI decide, "I'm going to close in, fire off my torpedoes, and then back away". Behind the scenes, the code actually supports this sort of thing, but it generally doesn't work out in practice - combat situations are too fluid and a plan has to be constantly re-evaluated to see if it still makes any sense in the current tactical context. It's often easier to not have a plan in the first place.
One sort-of exception here is the AI controlling phase cloaking, where the AI decides to go on an attack run and unphase behind the enemy ship. It does this by setting a few flags in that shared object, with the movement module having some explicit support for handling the movements required.
Also, if I may steal just a drop of precious time from Alex's schedule - do you regret your choice of AI paradigm now that you've had to deal with it for years? Or if not, do you have any tips to deal with its downsides?
I'll go with "no, I don't regret it", mainly because the paradigm evolved along the way to fit what was necessary. The original design was more focused on plans - a sequence of maneuvers - which, as noted above, didn't work out. If I had to rewrite it from scratch, it'd be about the same thing but a bit cleaner and better-performing.
The main thing, I think, is to not worry about the code unless you need to worry about the code. What I mean is, 95% of the code in a game can be total spaghetti, and that's fine. It just needs to be modular enough that you can replace the pieces as needed. Not that I'd advocate writing bad code on purpose, rather just not tweaking code past the point where it works (which includes performance, etc).
I'm not sure the above is useful advice, since so much of that is feel and experience. I do remember, way back in the days before Starsector, being concerned about the "quality" of the code I was writing, getting the design just so, etc. And I can't even say that was entirely useless, since those things require some thinking about to understand. And then you have to forget about them most of the time.
Eh, that ended up being a bit of a tangent. To bring it back on point somewhat, I don't think the ship AI is held back at all by the way the AI code is structured. It's flexible enough that anything practical could be hacked-in on an as-needed basis.