Yep - welcome to spandex! You want your bike shorts to fit like skin. That gives you muscle compression which helps your muscles not to fatigue so quickly. Compression also helps with recovery.. There's actual medical evidence of that, although I don't know the science of it, I know it works!

Listen, you are a size Medium, mid-50's, riding a bike... you are fabulous! Don't worry about it!