I'm kinda voting for the saddle. When I was 1x and 2x my biggest problem was the saddle, so I'm biased.
I wore the cheapest Nashbar-special shorts, but once I switched to a Terry Sport saddle (which they don't make anymore) the world was a better place. I still have one pair of my 1x shorts, which I wear as liner shorts. (they are waaay to big now to wear alone)
Nowadays I ride on a Brooks B67, and want a Brooks B68 for my other bike.
My first thought it always going to be the saddle, so take my 2 cents with a grain of salt!
"If Americans want to live the American Dream, they should go to Denmark." - Richard Wilkinson