Probably a combination of all that you have listed.

Distance--keep riding.
Core strength -- that's easily addressed. Get working on your core. You will feel better all around. I prefer Pilates, but find what works for you.
Saddle--sounds like it works okay for now
Fit--maybe. If you can afford a pro fit, you might benefit from getting dialed in.

I'd focus on the core strength. Along with that is stretching, of course (like yoga, and not when you are "cold").