If the shorts feel ok everywhere else, you could try removing the offending part of the stitching.

If you go to a fabric store and get a seam ripper and some Fray Check you can pick out the bad stitches and then dot the ends of the remaining seam so the rest of it stays put.