Well I don't have those shorts, but I've got to chime in because that's just weird. I take a Medium in PI and a Small in Terry.
That IS weird. I wear mostly Pearls, and they're all size small. The Terry Bellas (which I hate, BTW) are mediums...I tried smalls and could barely get them on!