Might want to get a rear rack that has different connect/load options and/or different bags so that you can hook up further back to avoid any contact with clothing. I have Ortlieb Sport Packer Plus front packs that I load to the rear rack mostly, and a set of the Back Roller Plus bags. These bags are pricey but they are waterproof, lightweight and adjustable and they won't wear holes in your shorts.

Click image for larger version. 

Name:	IMG_2473.jpg 
Views:	444 
Size:	125.4 KB 
ID:	16671