I had pneumonia about 6 years ago and also as a teen. The last time it happened after I got a sinus infection, went on a tour in Vermont for a weekend in coldish weather (October) when I knew I should have stayed home (we had already paid). I felt better in just a couple of days, but when I tried riding after 2 weeks I turned back after 5 miles. I was weak and couldn't breathe. I remember going on a ride on Veteran's Day, maybe a month later and I still was tired out and weaker than usual. The cycling season ended a little after that, so I moved indoors and was fine. No problem nordic skiing in December, etc.
About 15 years ago I had the flu in March. It took me until May to recover aerobically. I was coughing like crazy for almost 8 weeks after the fever broke. I was teaching aerobics at the time and I couldn't go back to that until the cough was gone. I did swim, slowly, though and it didn't seem to bother me.
So, really listen to your body.