I get that sort of pain toward the end of long rides. It's mostly because my core is weak, and I'm very right-sided. I end up putting much of my weight on my right side (which is why my right wrist hurts right now and the bar tape on the right side is all messed up!), so those lower back muscles end up overworked.