Just my .03 worth - read a journal someone wrote after following Lance Armstrong on a training ride and the two things that stuck the most with me were: Lance spends time every ride riding with no hands (is that the right term), and he rides on every type of surface including dirt/gravel. When he was asked about both quirks his answer was - anything he could do to improve his bike handling skills and his balance made him a better rider when in a race situation. (think Lance riding across the plowed field during that horrible crash in the TdF few years back)
While not a racer - it made me realize that that was an excellent premise to ride by, that way a clutch situation becomes "just another type of ride". So why not ride your road bike on the dirt road.......it could just improve your handling skills if your road bike ever ends up in dirt during a "situation".
As for buying a house to suit your riding?.....why not - it's as much an important part of your life as anything else you're using for your house buying criteria!