My DH naturally evolved to be being more of the kind of rider that I am. In the interim, I got faster, he's fine with rides at my average, and if he wants to hammer he does it on his commutes. I also found a group of like minded riders that are a sub group of a club known for more hard core rides, as well as finding a couple of other individuals to ride with. That being said, I do most of my riding with DH. He pushes me once in awhile, still, but only when I feel like it!
DH did not really enjoy nordic skiing like I did, and for a few years was not as skilled as me. When he got new skis which allowed for better control, he improved and now we are fairly equal. It was good for me to be on the other side, for once. On the other hand, I do more all around fitness stuff. If I didn't push him, all he would do is ride.
I am trying to think back to when DH started riding and I was still a gym rat, who was burned out and gaining weight. He generally scheduled his rides at times when I was at the gym, or doing something else. We still did other things together, and I never felt jealous. You need to talk to your DH and get your physical health under control. My view of this changed when I had a mysterious illness 4 years ago; I had to give up some control and enjoy taking walks and slow rides. I spent one summer doing a lot of farm stand rides, to the tune of 300 miles, and not worrying about my speed on road rides. I still go through periods when I can't do as much as I want, but because my sports activities are a bit more varied now and DH shares most of them with me, I am fine.