Hmm. First of all, I readily grant that the dialog about whether you want to retreat is annoying
I'm not sure whether expanding the area where being near a border affects gameplay (i.e. via gradual pushback) is a good idea, though. The Map border isn't the kind of mechanic you really want to build on, but rather want to keep as unobtrusive as possible - it's there for housekeeping reasons, not because anyone actually wants it or is excited about it. With that in mind, I'm definitely open to suggestions for how to make it more unobtrusive.
As to removing borders altogether, I don't think that's practical. If you consider in detail how things would play out (or, indeed, mod in a very large map for small, no-objective engagements), I think you'll start to see some of the problems. Just
one example: what's likely to happen is for a few clumps of smaller combats to drift away from each other - leading to a situation where it either takes you minutes to move from one clump to another (once the battle there is finished), or to where you couldn't catch up at all due to being in a slower ship.
Then there are some nuts-and-bolts difficulties in having something actually be unbounded - not insurmountable, but not something I'm inclined to pursue, given that I think it's going to cause more problems than it solves. Conceptually, having map borders doesn't bother me - the way I think of it is ships retreating off it simply engage their travel drive, and thus can't be caught up to.
In practice, yeah, that dialog box is an annoyance, and being able to corner ships isn't exactly good. But, having objectives means that battles tend to take place away from borders, so the actual gameplay impact is reduced. As I mentioned earlier, I'm open to other ideas for further reducing it - without introducing other issues, though