It sounds like you need a bike fit. It's normal to be a little sore after the first couple rides, but not like that. And the numb fingers on rides of that length are definitely not normal and are concerning.

Hope you can figure out what is wrong and that your bike can be tweaked to work better for you. I never have any pain once I'm off my bike, and I'm only doing short rides a couple of times of a week this time of year myself.