Does anyone know if leaving the valve caps on inner tubes increases the likelihood of damaged valves?

I seem to be experiencing a bout of flats caused by problems with the valves, rather than p*nctures.
I'm using long valve presta tubes, as my wheels are quite deepset. I'm really carefully when pumping up the tyre (ie making sure the valve isn't bent when unscrewed, etc and use a track pump to inflate).
The problems seem to occur when doing a lot of hill work/steep, windy bendy descents and have heard that leaving the caps on can cause damage to the valve as it pushes against the cap.
I make sure the tyres are aligned properly in the rim, that the inner tubes aren't pinched or the valve skewed, but have now had 4 flats in the past 6 months from the same problem.
Does anyone have any ideas? It's not the base of the valve that seems to be the problem, rather the whole thing fails. And I've tried 3 different types of tube!