Hmmmm,
Well, if you and your DH are riding *together* than it's nice if you actually ride together. Otherwise, it's just two people out for a ride who happen to be on the same road.
When I ride with DH, there are times when he wants to motor up a hill and I don't. He waits for me at the top.
On our group ride, there are wait up spots. If the group splinters, we'll wait at a light, even if we go through one (or more) light cycles just standing there. It's to keep the group more or less together.
You both have to clarify what the goals of the ride are. If they are togetherness, then ride together, since his going ahead and looking back is plainly bothering you. If he wants to hammer and you don't, either don't ride together, or have him wait for you up ahead at pre-determined points.
But communication is the key.