I've owned Starsector for a long time (since Starfarer at least), recommended it to plenty of friends, and I'm sort of ashamed it's taken me this long to join the forum, despite reading it for ages. I guess I never consolidated all my thoughts until now, when the game is together enough that the little gaps get to be more noticeable.
1. General target priority selection. While doing this per-weapon would be extreme (although I would love something like "tell an autofiring missile group to not target fighters/frigates), telling individual ships something like "focus enemy fighters" or "focus on avoiding combat" would be great. I'd love a button that tells a Hyperion or Phase Ship "don't take any risks and just skirt the enemy, making them have to cover their back." Officer personalities do this somewhat, but there isn't a personality like "defend friendly ships from fighters/missiles" or "stay away from the front lines and hunt carriers/missile cruisers," while the way in which AI personality would combine with these general ship orders would add another layer of depth. Again, some of this can be done in other ways (load a ship with PDs, throw on a steady/cautious officer, and tell it to escort something), but for something that's such an intuitive role on the player side, it requires a lot of effort, compared to just checking a box that says "provide PD support to friendly ships that lack it." Ultimately, options like this are the only way to create fleets with complex teamwork in the long term.
2. Finite system use aggressiveness. To give a very simple example, let's say you take a Gryphon, and give it a Squall (large), 2 Pilum (medium) and 3 Annihilator (small). A checkbox saying "use the system liberally(when the ship first decides it needs to fire a missile it doesn't have a full volley of)" versus "use the system conservatively (wait until all missile batteries are below enough missiles to volley before you reload)" depending on whether you want to set it up for striking or barraging, while still giving it some missiles types to cover it in the other role.
3. Automatic retreat thresholds, either at a "deployment time remaining" amount or at a "CR remaining (nullified if deployed at or below this)" amount, or just straight up "out of ammo/missiles on non-regenerating weapons." While something like dagger wings do this naturally, I think it'd help to be able to design genuine one-shot strike frigates. As a player, I'm capable of using a 4-reaper Hyperion to teleport behind and one-shot many large ships, including Onslaughts. It's disappointing there isn't a way to tell the AI "get in, find a big target, torpedo the crap out of it, then skirt the battle until you're low on CR, then retreat" even though, for a player, that's a very simple strategy to understand.
You have to draw a line before you're just making people program their own AI in this game, but when there's this feeling of "if you want something done right, you've got to do it yourself" popping up with certain roles, it's worrying. The AI is perfectly trustworthy on a stand-off or pressure focused capital ship, a carrier, a missile barrage cruiser, a harassment frigate (in small engagements) and yet falls apart in some roles (a player in a Monitor can build it for flux and just rely on turning on fortress shields and retreating, but the AI will always turn them off for short moments and end up getting killed, a player can build a "bombing" phase ship, but the AI will never be as protective of it as a player is.)
4. Even if all of that would just add needless complexity, I think that, especially with the advent of officers, there's some simple idea that would do everybody well. Default orders, especially escorts. So you tell a ship "by default, escort this cruiser" or just be nonspecific about it - just say "by default, this capital ship gets a medium escort" and it would require use of CP to remove it in the actual game. You could also change your default orders for one battle before you enter, same as you can press "Transfer command," you'd be able to press "Change battle plan." If you want to have a fleet built around a simple strategy, you need to build up a CP pool and open every battle with a series of identical clicks - carrier rally somewhere near the spawn, PD destroyer follow this assault cruiser, etc. It would be trivial, useful and intuitive to make that strategy a default.
Beyond the player being able to use this, it would allow making AI fleets with clear flavors and preferences in their battle strategy - maybe one faction uses its fighters to capture points, while another uses its fighters to escort capital ships. One faction has all of its capital ships sticking together while the smaller ships skirmish freely, and another has its larger ships all spread out with their own escorts of smaller ships.
-----
I have some experience with game design, a large part of which isn't about having good ideas, but having practical ideas - something quick to implement, easy to get feedback on, and user-friendly in both concept and interface. Hopefully at least some of this helps. I wouldn't want to go down the road of making this a "programming game" to the degree that new players are just told "go download this script for your AI frigates and let them do the work for you," but I think some of these changes would allow players to builds fleets in ways they otherwise couldn't, because there are certain roles they don't trust the AI in, while at the same time letting enemy AI get a bit smarter. How many dozens, hundreds or even thousands of battles open with some enemy Hound or Vigilance rushing forward and immediately getting torn apart by a Tachyon Lance, Hurricane MIRV or barrage of Hypervelocity Drivers? It's not that one under the player's command would do differently, it's just that the player would have it sticking close to one of their big ships. If it dies, it's at least taking resources that would've otherwise been spent on the ship it's guarding, rather than just being a "free" kill that the enemy can vent the flux from.
If the enemy had to use CP, then those 30-ship-fleets with 15 frigates would barely have enough just to get all of their frigates escorting something. With default orders, everyone wins. Except the Hegemony, because guns are stupid and lasers are cool and they should feel bad.