I guess that I'm unusual in that my husband and I ride together. It didn't start that way, but he was unbelievably patient with me and before I knew it, he slowed down a bit and I got a bit faster. I still ride alone for short week day rides and so does he (commuting mostly). I ride with a friend once in awhile and do group rides, but not so much anymore on the group rides because the lack of riding etiquette gets to me.
I know I would continue riding if my husband wasn't here, but I don't apologize for wanting to be with him most of the time. Life is too short and I feel unbelievably lucky to have a spouse that i would rather be with than anyone else.