So I started an Act 1 game, progressed about 90% of the way through it (from start to Leoric's Hunting Grounds, doing every zone and optional cave), then opened up the map and started to scroll around a lot. After looking at most of the stuff on the map, my FPS went from 50 to 3, or 2, or 1...

It fixed itself after about a minute of me doing random things in town (i didn't want to lose my NV!)

ATI 6800 graphics card, latest 3.14 (?) drivers.