Though this is probably more useful when solo-ing, I generally always swap on haunt before a boss fight and pretty much used haunt alone to kill diablo while I ran in circles around him lol. Though yeah outside of boss fights I don't find it as useful.
On a side-note, I do feel like where I am in the game(midway through nightmare) I don't do much beyond splinter dart and dire bats for regular mobs which is depressing. I forced myself to play with the other primary/secondaries but have found that they are sorely lacking in relative DPS... Bats have plenty of aoe and splinters clear stragglers if they were outside the path of your bats. If enemies survive a mana bar of bat spam + spirit walk mana regen you need better gear. Wish I found more appeal in the other skills.