If you like riding in general, then I would start training alone and with your husband when he can take it easy with you. Ask him nicely to back off with the comments. You understand what he's trying to get you to do, but you're still learning about how to implement it for yourself.

If another couple or a few friends invite you out for a casual ride, then they need to stick with you. I think it's ok for the group to split up on hills, because everyone needs to go at pretty much their own pace. Even if the faster people are taking it easy, that may not be easy enough for the slowest. But then there should be some periodic regrouping.

It doesn't sound like you're trying to do some competitive group ride (and even then, many will have some regrouping points along the way, especially this time of year). So, if they're going to ride with you, they should ride with you.