I think part of the issue is ships in general will turn to fight enemy ships that are engaging them from a flank. Especially if said flanker is smaller or locally less superior as calculated by the AI and there's no larger threat nearby. This is the same reason enemy AI ships will turn towards the player ship when it's off isolated from the rest of the fleet doing a deep flank solo, trying to hit ships from behind.
Engage doesn't override the fundamental AI, not nearly as much as say an Eliminate or direct retreat order. It's the "fight, but don't die while doing it" command.
As it says, "Order your fleet to engage the target and any nearby enemy ships, without exposing themselves to undue danger". You could imagine it adding an additional "weight" to move in the direction of the target, but that weight is not very strong compared to other things it is being weighed against like nearby enemies. Retreat does the same - it adds an additional weight towards the retreat side. Eliminate and direct retreat I believe start to make more drastic changes to how enemy ships are weighted, which is why your ships will dive deep into enemy lines and get themselves destroyed with those types of orders. They're essentially ignoring most of the enemy fleet for position purposes. The weight that Engage adds also doesn't change with the number of times you've issued the command. The AI has no concept of command history, and Engage is essentially a toggle. Given the exact same command 3 times isn't increasing the likely likelihood of the ship behaving you way you want. If it's on, it's on.
Imagine you've given no orders to a Falcon, and a Tempest engages it from a flank. The Tempest gets into firing range, and the Falcon is going to turn to bring weapons to bear, and fight it. The Tempest perhaps runs up it's flux and starts to back off. Now the Falcon is nearby a high flux enemy it's been engaging with, and instead of letting it just run away and vent, tries to pursue to finish it off. Which is presumably the behavior most players would want in that case - finish off high flux nearby ships. Problem is the Tempest is too fast, backs off while being chased, and gets it flux down, and dives back in. Thus you have a frigate distracting a cruiser an entire fight.
Now imagine that same situation, but you issue an Engage command to the Falcon to fight another target elsewhere. Since the underlying AI doesn't change, and it's got a clear and present danger here, it is going to follow the same behavior. It's going to do what it thinks is best for it locally to not die or take significant damage. That generally means engaging nearby enemy ships instead of ignoring them and letting them flank the ship. If there's no other pressure, the Falcon will follow the engage command, but if the ship itself is being engaged by an enemy ship (imagine the enemy ship has been given an engage command against your Falcon), it's not necessarily possible to extricate itself. It's going north because it's consistently winning the flux war and the harassing enemy ship has to back off briefly, but if the Falcon doesn't finish it and turns towards the cruiser elsewhere, the enemy ship is going to be right back on it in 5-10 seconds and hitting it from behind, forcing it to turn back again. The issue is in the AI's assessment of the Falcon's ability to finish off the target quickly and the default underlying behavior of trying to finish off "inferior" targets. Or in the assessment of how dangerous the target is and whether it could be ignored, which the AI is understandably not nearly as good as a human at doing.
If the not important ship the Falcon was engaging was slower, and the Falcon had no flux issues, I presume it would finish off the unimportant ship and move on. The fact it was being distracted long enough for 3 Engage orders to be issued suggests to me the other ship was faster (or potentially a Monitor). If the other ship is faster, the Falcon can't control the engagement, the enemy ship does.
As it is, there are plenty of posts where players are unhappy with how unaggressive ships can sometimes be against an overloaded or high on flux ship. So tweaking how much ships backoff for the overall AI is potentially going to have undesired side effects for more general cases. Use of "stronger" commands like eliminate, and applying selective avoid commands, which change how ships are weighted and so forth are likely to be more helpful in situations like this. Of course, the physical capabilities of the ships may still result in a situation that doesn't go as planned. Avoid doesn't work when your ship is an Enforcer and the enemy is an Aurora plasma jetting in.