I'd say anything strong enough to move your bike to where you didn't plan it to be is too much wind because it could push you in front of a car.

I can't imagine riding in 40mph or 50 mph winds, I rode once in 25 mph gusts and it was scary enough.