Well, the way you are feeling is normal for someone as active as you are. Do not beat yourself up about feeling angry or depressed. But, you are right, you have to think of your baby. Given your history, I would not go against the advice you got. If you had not had a premature delivery already, my advice would be to find another practitioner.
I say this with some (old) experience. I went into premature labor at 30 weeks. It was the result of a stupid bike mishap, where I leaned over too far, my balance was off, and the down tube shifter hit me really hard in the abdomen. It was just a leisurely ride in the neighborhood and I had been doing aerobics 4-5 times a week up until this point. A week after this accident, I went into labor and was put on "house arrest," couldn't work, or really take care of my 2.5 year old. I had to take meds every 4 hours. I was furious and so upset that I actually took a plate and hurled it to the kitchen floor in a rage. That scared me, because my toddler saw it and went hysterical. My DH called my mom, who came from another state and stayed with us for 4 weeks. My son was born at 36 weeks and was perfect.

I would take it as a challenge. Work on your swimming skills and walk quickly. It's just a few months, even though it seems awful now. It's not worth the risk.