I am not a racer, but I had similar issues, stemming from 3 years ago, when I was "supposedly" diagnosed with fibromyalgia. I say supposedly, because that's what my doc decided when he couldn't figure out anything else. I've always been susceptible to over training, so I am used to watching my energy level, etc. Otherwise, it's a series of endless little illnesses that will knock me out for a long time. Or, it turns into pneumonia, bronchitis, etc.
After not really riding at all from September to December 2007, I started the 2008 season with high hopes. But, I found my average was at least 2-3 mph slower than it had been and things that weren't hard for me before, were really hard. It wasn't every time I rode, but sometimes, even a 15 mile ride would cause a "reaction" and I wouldn't be able to ride for a few days. That all started making me feel psychologically worse, like OK, I am just getting old. I still rode faster than my "slow" friends, but it didn't make me feel better.
In 2009, I pretty much stopped looking at my computer, for average. I concerned myself with miles. I rode my hybrid a lot, doing farm stand errands. I still rode over 2500 miles last year, but mentally I had a better attitude. I also spent all of last summer doing yoga/meditation and when winter came, I did a lot of x country skiing, challenging myself with that.
I have found that this year, I seem to have my "mojo" back. My average is close to where it was in 2007, but I still ride slowly, on purpose, at least 1-2 days a week. The other days, I do intervals, or try to keep up with DH. I know that a lot of this is in my head, so that's an aspect you have to consider. Be kind to yourself!