Cut the piece of velcro with the hooks so that it's short enough to be away from your legs, but just long enough to hold the bag in place. The straps that hook to the rails do most of the work holding the bag on, so you don't need much velcro on the seatpost part. It'll keep the edge from snagging on your shorts so much.

If you have the problem on both sides, then look for a different attachment system.