Maybe it is an allergy, but it just takes having the shorts on longer for it to affect your skin? Or could the shorts be rubbing your skin, irritating it after wearing the shorts for an extended period of time?