Hmm, what you said about going slower and hurting more, and then going faster later and not hurting struck a chord with me. I've noticed that when I ride with some good friends who ride slower, I seem to sit a little differently, and put more weight on the downward motion of my feet, so they go numb more quickly than usual. And I lean differently, so my arms get tired. When I ride a bit faster, I tend to sit a little differently, and everything's more comfortable.

Maybe you were sitting heavier to go slowly?