I think there are two different types of issues I was kinda addressing at the same time here. One is making decisions about immediate engagements, and the other is making a 'long term plan'. Some problems, like overextension, and cautiousness in overwhelmingly one sided situations, are issues about immediate engagements. Some problems like chasing down frigates, or deciding to help allies, are more about coming up with a 'plan'.
In terms of immediate engagements:
(I think for humans this type of "density" is a first-order heuristic - how we narrow down the field of possibilities to look at things in detail before making actual decisions.
I don't know if I agree with that. It seems to me like the player does make decisions like 'There are too many enemies near me, I need to back off', or 'there are lots of allies around, I can send it' directly from their assessment of fleet density. I think the best example where the current AI falls short is a frigate near a radiant. The AI frigate has no idea it is in danger until it dies instantly to the radiant, but the player knows that they are in danger before the radiant actually shoots and backs off to avoid that situation.
I'm definitely not suggesting to replace any of the current AI decision making processes (which are definitely valuable), just add an additional factor to them.
When you have the AI make decisions based on information like that, it's bound to get things badly wrong in terms of specifics; you'd end up with "why isn't it engaging" and instead of the answer being a relatively clear "flux is too high or it would be flanked if it did" it would instead be "the invisible density heuristic tripped a threshold because a fighter wing or a frigate 3000 units away moved fractionally closer".
I think the sensitivity to small movements of ships shouldn't be such a large issue, i.e. if one fighter wing moving closer is enough to move you over a threshold, you were already
right on the edge of needing to back off, so it shouldn't seem unreasonable. I feel like 'it backed off because there were too many enemies close by relative to the amount of friendlies' is a pretty intuitive explanation, very similar to 'I was getting flanked' IMO. I can definitely understand that the details of implementation could get messy though. It just seems clear to me that this is the sort of information the player uses to make decisions that the AI currently is not good at.
In terms of long term plans:
Incidentally, in 0.95a the AI had a behavior where larger ships would try to group up and avoid getting sidetracked by enemy frigates. The end result of this was a bunch of confusion over the AI "refusing to engage".
I think this not working is sort of getting at the problem. The issue isn't really about 'sticking together', or even proximity to specific ships, it's about the AI not having any plan for what it should be doing. Of course the player can make those plans, but I feel like it could be valuable for the AI to have a decent plan on its own. That seems to be inline with a lot of other design decisions that lead towards the player being able to just pilot their ship if they want.
In terms of existing game mechanics, I think what the player wants is the AI to assign itself an engage order on a 'relevant' enemy ship. The problem is that 1: engage orders kinda don't work very well (I find I need to assign and cancel eliminate orders to get the behavior I expect from and engage order), and 2: the AI doesn't know what 'relevant' means and that's where the fleet density idea comes into play. I think this sort of logic already exists in the game in the system that automatically distributes ships between capture/defend orders.
The clustering idea is a way to dynamically identify points of interest without the player needing to give a ton of orders like they do now. I feel like giving defend orders on ships is running into the same problem that the 'stick together' solution to chasing frigates did. The player doesn't really want their ships to stick together (although that might be the result sometimes), they want their fleet to be distributed well to engage the enemy, and they have to be juggling orders to keep up with the dynamically changing enemy positions which could be described as 'babysitting' or possibly 'herding cats'. It would be nice if the AI had at least a basic understanding of where it would be valuable to go, and the player can work from there.