It appears that sometimes when changing passive skills, the skill doesn't take effect. I've been able to reproduce this in town and out of town, even when the cooldown for changing the skill has finished. It only appears to happen when you change more than one passive skill at a time.
Here is a generic example of how to reproduce the bug:
1) Open skill panel
2) Select a passive skill to change (for my tests I changed the 2nd slot)
3) Change that slot to the desired skill
4) WITHOUT going back to the skill selection page, click another passive skill slot at the top of the page (for my tests I changed the 1st slot next)
5) Select a skill for that slot
6) Notice that (for this example) the 2nd slotted skill works, but the 1st slotted passive skill never takes effect.
I was running around for quite some time before I noticed that some of my passive skills weren't working, so I imagine a lot of people are having this problem and might not even know it.
WORKAROUND: The current workaround appears to be to only change one passive skill at a time (and go back to the skills page before selecting another). All passive skills seem to work then.