The act 2 wasps that fly around and shoot the little bugs at you... they are the worst!
A Yellow pack, with Hoard, Magma, Missle Dampening, and Extra Health.
It has to be -literally- the most annoying pack I have ever fought. It shuts down both ranged and melee combat with that combination.
Annoying doesn't mean hard of course... here is pack combinations I find annoying, but not relatively hard.
Electric + Knockback: the other affixes don't matter, this is just annoying. The lightning bolts tend to cause knockback whenever they hit me, so it's a very slow moving projectile that I have to avoid because it shoves me 400 feet away from the enemy. annoying, but not difficult.
Knockback + Vortex: Something annoys me about getting knocked back, then immediately vortexed, then knocked back, then immediately vortexed. While this can kill me if paired with other affixes,
Fast: This just innately bothers me. It wouldn't bother me anywhere near as badly if we didn't have a movement speed cap.
Frozen + Jailer: This is mostly annoying on a Yellow mob if it's minions inherit the properties too. Permanently stuck in one spot, while getting frozen constantly.
The above I find annoying, regardless of how deadly they are, or are not.