My BF and I live apart, so most of our rides together are while we're on vacation, not training. He's faster, so he'll pull if he wants a workout, or he hangs back if he wants me to set the pace. When we're on vacation, we use the camera a lot -- he hangs back and gets lots of action pictures of me, or he goes to the top of the climb, takes landscape shots, and then get shots of me summiting.
You can do things to even things out. Give the faster rider all the gear to carry -- pump, spares, food, etc. Give him a heavier bike -- he rides a mtb or hybrid while you're on a road bike. Have him do an hour of intervals and then come and ride with you.
For 3 days, I get to part of a thousand other journeys.