How's your nutrition when you're riding. The flipping of the switch sounds suspiciously like you might be bonking. Have you tried eating/drinking some calories at that point (or before??). Otherwise, it does take time. I've lost a lot of fitness over the last year, and am working HARD to get it back...

CA