Kudos to you for holding your ground and realizing that it puts *both* of you in an impossible position to expect to ride together and keep it positive. At least with the dynamics as they are now, it ain't gonna happen.
Now you want to find ways to build back your confidence because it has been artificially damaged. Find a way to get out and ride and build up strength and endurance and confidence even if it's oh, a 4 mile circle around your house. (I just went and rode 2 laps around work - 1.5 miles each - just to get outside and clear my head and get warm 'cause the air conditioning has kicked in here!
(Can't give advice on relationships except for not setting up disasters... I'm a singleton )