I assume you are talking about pve. I understand your problem, but what usually happens when you have several targets together is that you don't really care which one you are targeting, you just use your aoe rotation to deal damage to all enemies at same time.
For some classes (like enhancement shaman) you better target one that is NOT almost dead do apply your aoe rotation, but as a general rule, it doesn't really matter.
When there is a real need to kill in a specific order, it is not usual to have more than 3 targets to pick. At least that is my experience.
My aoe "rotation" includes putting dots (sometimes multiple dots) on each and every target and keeping them there. There is no magical button that puts dots on all of them for me, so I have to target each one as the dot is running out and refresh it. That means I most certainly care about which one I'm targeting.
If it's not possible to stack the enemies in a raid-frames style of grid, it's not possible. However, I *constantly* fight with the absolutely terrible targeting system in the default UI. When there are 5-10 guys on screen and I'm trying to keep dots on them all it can be impossible to do something as simple as running through each one and casting one instant-cast dot.
The default tab targeting is bad at best, and gets even worse. I've had 20-30 guys standing on top of me beating me, and tab does nothing at all (so I'm still selecting a corpse or something). Or it selects a guy behind me or just out of range (so I can't even cast an aoe threat generator). Or it selects a guy who's not even in combat (so now I have even more guys I can't hit on top of me). On occasions, it even selects a guy who's behind me, out of range of any of my spells, and not in combat (which isn't the worst case, but. . . how?). Sure, mostly it sorta-kinda cycles through the mobs in front of you, with some kind of retarded accidental-click protection that I can't find any way to disable (how I hate that "feature"), and is reasonably ok. But then it goes to hell and I've spent 5+ seconds and just decided to /tar <enemy> on one of two mobs attacking me, because it worked better. Or died.
Trying to click an enemy (or player) who is "behind" another enemy can be impossible,and the hit boxes on many bosses are just ginormously bigger than the actual boss, making the problem even worse. I spent a good 30 seconds in HoF LFR last night trying to click past the boss and grab a player out of the tornadoes with Leap of Faith, and finally just gave up. I suppose I could have turned on friendly plates and tried to click those, but trying to click the player model itself wasn't going to happen.
Trying to click a nameplate usually results in a different nameplate being selected than the one you wanted. There's some kind of z-ordering filter, so if you click enough times you'll eventually get it, but if you take your time, the mob has gotten the cast off before you can interrupt or the friendly is dead before you actually select them, and if you try to rapidly get to the one you want it's really easy to click once too many times and end up needing to start over again. Furthermore, on many of the bosses and large trash mobs, the nameplates are 9 miles above the ground, so you can't even see the nameplates to try targeting them anyways. Unless you stick the camera under your feet, which is just annoying.
Obviously, I manage to deal with it, and in most cases the system works well enough, but it is hardly ideal. Many, many times, I've just starting mashing buttons on my tank until something happened because that was more predictable than the WoW targeting system (to be fair, I'm usually mashing buttons related to targeting functions while mashing aoe abilities in the hope that I will coincidentally aoe something while it's targeted). On aoe fights, I've often just applied dots until tab either stops cycling targets or brings me back to a mob I've already dotted (usually after 3-4 mobs out of 10-20), then just using hellfire/mindflay/whatever and not caring about my dps because it's just not worth the effort.