The above the neck rule is a good one to follow, but allergies can and do make you feel weak and sick.
I am the queen of riding when I shouldn't and it has caused more trouble than it's worth. Bronchitis, pneumonia, etc. My immune system is not that great and my advice would be to listen to your body and take a day or two of rest.
I wish I had done that six weeks ago.