Get a trainer for your bike so you can ride early mornings/when he's otherwise occupied, but with you? Also get some Spinervals workouts so you can get some good, focused workouts in?

If you don't want to do that, I might swap one of your climbing rides for intervals (others have better descriptions of how to do that than I do - but the idea is to really push the pace for short periods of time, recover for a little while, rinse, repeat).