i'm on your husband's end of things...i'm definitely a lot slower than my DH. whenever we go out for rides, i know i'm holding him back. but he claims that riding slower with somebody is better than riding faster alone. i do encourage him to do rides at his own pace, and there are certain rides that are tough for me (or some of his friends i can't stand), and he does those alone too. (i'm not much of an alone rider...i get bored). however, due to the fact that he has spent tons of time riding with me (and getting me on a trainer), i've noticed amazing improvement. so definitely have a ride or two for yourself, but you'll be so happy with your DH's improvement after you help him for awhile, and your alone rides will be less frequent
"humanity has advanced, when it has advanced, not because it has been sober, responsible, and cautious, but because it has been playful, rebellious, and immature." - tom robbins