Some people are very threatened by the idea of getting from Point A to Point B without burning fossil fuel.

If someone asks again, why don't you just tell them the truth, that cycling a short distance in street clothes is fine and convenient, but over longer distances (like 15+ miles) it's not only very uncomfortable, but likely to ruin your nice street clothes? I wouldn't feel like I was discouraging them from starting to ride, since a brand new commuter isn't likely to jump right into 15 mile commutes in any case.