Maybe you just need to ride separately sometimes, so that you can do your cruising-along thing and he can do his pushing-himself thing.

My husband and I have a similar situation, although it's more about degree; he and I disagree about how much I'm pushing myself!