If the rubbing is on the outside of your shorts, then +1 on the saddlebag. That's a very common issue. It doesn't even really have to be catching on the velcro, just the edge of the webbing. A small piece of matching electrical tape on the strap will solve the problem.

Another possibility is the saddle clamp, if you have your saddle set very far back on the seatpost. To fix that you'd need a different seatpost with more setback (assuming the saddle is in the position you want it). Last possibility is the saddle itself, especially if it's a tensioned leather saddle. Check the outer lower edges for burrs that you can gently file off.