Each time this happens, Activity Monitor has shown a process named 'Agent' running, taking a very small amount of CPU, but still more than any process other than D3. I don't know what Agent does, but hunting on the forums shows it does have something to do with D3.
I have taken to killing this process through Activity Monitor. Each time, this has resulted in my FPS spikes going away, and FPS returning to whatever the steady state is for the area I'm in.
I hope this information is useful to someone!
It's entirely possible 'Agent' is important and I shouldn't be killing it. So use at your own risk ;)