I think you should go out there and give it a try. What's the worst that could happen. You might have to walk up the hill. So what!
I never stand going up hills I always sit and spin because I don't want to wrench on my knees.
I have been riding again for about 2 years now and I don't worry about my speed. I worry about getting through the ride and not killing myself.
I always feel bad if someone has to wait for me too, but I have learned to ride by myself and not worry about everyone else or trying to kill myself to keep up. Just make sure you know the route and how to get where you need to go and have a cell phone or something with you for emergencies.
If they say that it is a no drop ride, then they need to expect that there will be some that will be slower and someone will have to hang back with.

Seems to me your husband should be encouraging you to join him and he should hold back with you, and help you along, since he is experienced on the ride.