my bf will do this sometimes, though usually he tries to stick by me. he'll get about 10 ft ahead of me and tell me to hurry up. my problem is he's been racing and riding for years and years and i just started. so i know i'm not as good as he is. i'm happy when he does ride with me, makes me go faster even if he's getting ahead and looking back (which does get annoying and i usually scream for him to keep going and i'll catch up to him sometime).

at least your DH didn't power up the hill infront of you, get off his crossbike, pick it up, run next to you for a bit while you barely passed him, and jump back on the bike and ride past you. i know he did it in good fun but i still had to give him a pretty finger of mine (while trying not to show him i was actually laughing).