Going to town would be somewhat annoying just to swap skills.
Putting something like a 30 minute cooldown would make me NEVER want to change skills except when I'm leaving for the night.
A 5 second Interruptable Cast Time would make a good solution since you're not force to return to town, can't move, and if you're in battle you're bound to be hit within 5 seconds at least once. Maybe a limit it to damage while in combat so that lasting DoT's don't interrupt.
I immediately see a problem with changing skills in combat like it is because I know many people have keyboards with macro keys and people would be switching from pure AoE Builds to Single target, to Line target in an instant. Not good planning on Blizzards part if you ask me. Also in PvP the person with macros will most likely win against the guy without them giving an advantage do to Real Life tech rather then in game skill. Though pvp isn't a big deal for Diablo 3.
As for Out of Combat switching only, it seems like EVERY game that has this generally ends up with glitches of unknown enemies aggroing a player and then the player can't switch skills, can't heal, cant recover resources. That sort of thing.
5 seconds does not interrupt the flow of the game any more then opening the panel and switching out the skills (which at 60 level could take 2-3 minutes deciding) and tacking it on to the end of your selection doesn't interrupt flow too much but will take you out of the action enough that you can't fight AND switch. It also isn't too much time for your party members to wait, or allow them to get too far ahead of you if they won't wait.
A problem for this might be boss fights with large parties seeing as the boss can only concentrate on one person giving others the chance to change to Pure single target builds.