The only time I've ever had saddle sores on my actual backside it has been from shorts. If your shorts are loose the edge of the chamois (or any other seam on the shorts) can certainly cause problems - it can move back and forth and act like a little saw! It's a good reason that cycling shorts really should be tight. They shouldn't be able to be wrinkly when you are wearing them.

Even if the shorts are not particularly loose a badly placed seam can wreak havoc. One year the chamois in our team shorts had a seam that would actually cut your skin.... (I was not the only person who had the problem). When it was wet it was 100x worse - I actually had line of cuts on my rear once that looked like I'd had a close call with a shark or something