That happens to me because the shorts (actually capris) are a little too big. It seems manageable, though.