The issue might be you went to large.

I am 127lbs and I wear a S in all shorts/bibs, I couldn't imagine wearing an XL. I know when I have run out of clean clothes I have worn a pair of my husband's bibs before and I felt as if I was wearing a diaper.

My guess is you didn't need to go over a M