You want shorts to fit tightly, this cuts down chafing. So if you wear 4x you may want 3, 3x becomes 2 etc. If you were down to XL already I'd be sending you shorts. I have too many, some of which I've never worn.

You'll get there, keep riding and having fun.