I would tell him that it would be great to ride with him...but you can't do that pace. He should feel to go ahead and ya'll can ride home together...
You'll make friends along the way...and ride with a new group.
Last year for RaIN, Silver (the superior rider) told me she didn't want to push herself...but I did. I went on, found a spontaneous pace line of five guys (who's names I still remember) and we stuck together for 90 of the 160 miles. I had a blast...and Silver did as well with folks she met along the way.
Don't stress over this...but also don't try to "keep up with the Jones' " if it hurts
If you don't grow where you're planted, you'll never BLOOM - Will Rogers