"If you only knew how infrequently people were thinking about you, you wouldn't really care what people were thinking about you!" There's a saying that goes something like that.

It doesn't sound like you're worried about riding alone for the usual reasons - you're worried about it because you think if you're with your hubby people will think differently about you? Again, no offense, but I don't think people are thinking about you at all! Seriously! They're out suffering up the same hill. Remember - it doesn't get easier, you just get faster!