I'm kinda voting for the saddle. When I was 1x and 2x my biggest problem was the saddle, so I'm biased.

I wore the cheapest Nashbar-special shorts, but once I switched to a Terry Sport saddle (which they don't make anymore) the world was a better place. I still have one pair of my 1x shorts, which I wear as liner shorts. (they are waaay to big now to wear alone)

Nowadays I ride on a Brooks B67, and want a Brooks B68 for my other bike.

My first thought it always going to be the saddle, so take my 2 cents with a grain of salt!