We take it off. A couple of reasons....we drive a Prius and the Prius hates anything that gets in the way of its aerodynamics. Thus, even an empty rack can bring down our gas mileage (picky, picky car).
But, DH is a worrier and would need to adjust all the straps every time the bikes go on--so might as well just do it from scratch every time.
And, then, now that we've had the rack for a couple of years, it takes longer to gather up all of the bike "stuff" than it does to pop the rack on and off.
Just my 2cents.