OK... no, you DON'T sound like a terrible person... but I will say that riding alone, at least for me, is not nearly as fun as riding with someone else. That could be the case for the hubby too...

Here's the way I might consider handling it... ride with the DH for as far as he can go. When you return and he is done... go do a second loop, or however much farther you can go at your OWN pace... this allows you to ride together, you'll warm up for the faster pace, and you most likely won't be as frustrated knowing your ride time will be following your joint time. I actually do this when I ride with newbie ride friends... a ride with them is never at my speed or at the distance I would go... so I make sure they enjoy it and then head out for round 2 for ME!