I'm the same way with boy shorts.

It can't be that you're supposed to sit all day long.

It must be that our butts are too muscular. Yeah, that's it.

As to sizing, I wear a small Ibex brief where I take a medium in just about all other underwear. Generally if a brief rides up on me, it's because it's too large.