There must be an opportunity here for a "beat the velcro rub" product for cyclists. Dunno what it is, but it must be there...

We've had the same problem in our house too