Huh, well I actually decided to test and it turns out that I was wrong: The RX 580 is not even close to being a bottleneck, so it makes more sense why I didn't see any improvement with it. I'm totally CPU bound. Running some of the larger missions as tests, I'm clocking 100% CPU on one core, with GPU flickering between 0 and 30% - hw monitor can't seem to make up its mind, which is odd, but its not building up enough heat to spin up the fans. (Could this be that the openGL is so bad that it can't even run hot?) In either case, CPU is pegged close to 100% on one core so.
So, despite AMD OpenGL drivers that indeed are terrible, I can say that for my system the GPU is about 3x more powerful than it needs to be.