I'm the slow one...
I ride a C-D place in our club's designation (about 12-14 mph, depending on the terrain and how hard I feel like pushing), and hubby's more like 17mph average. He goes on *his* separate rides, and rides with me sometimes. I go on *my* separate rides. When we do ride together, what works best is for him to have done a "fast" ride, and be a little tired when he starts riding with me - that way, it's more of a cool down/recovery ride for him. Of course, he still zooms ahead on occasion and waits for me to catch up... I probably ride more regularly than he does overall, but he's still faster.
I would be concerned with your always riding at his speed that you don't get enough of a workout and won't improve your skills. Wanting to challenge yourself *does not* make you a bad person. It needs to be fun for you too!
Most days in life don't stand out, But life's about those days that will...