If my broom misses, or you avoid the swap, I lose. No matter what, I lose.
Even if it hits every time, even if I play perfectly, I still lose sometimes.
Forced swap moves are fine, stop crying.
A forced swap that goes 1st is guaranteed free damage at no cost. If it misses, you're not really far behind (his move might miss just as well as yours). You just don't start with a big advantage in damage.
I think swap moves are cheesy personally. In their current implementation. But that's just me.
Not saying it's good or bad; just pointing that out.