The issue might be you went to large.
I am 127lbs and I wear a S in all shorts/bibs, I couldn't imagine wearing an XL. I know when I have run out of clean clothes I have worn a pair of my husband's bibs before and I felt as if I was wearing a diaper.
My guess is you didn't need to go over a M