We left ours on the roof all winter, even, because we never got around to taking it off. Definitely impacted our gas mileage, but we drive infrequently and short distances, so we don't feel that too intensely. On the other hand, it also allowed us to meet some friends - when DH had the winter tires swapped for the summers at the tire shop, our new friend ran over to ask him what types of bikes we ride - they are actually our neighbours, our backyards are just a few yards away from each other! And, DH raced with him as his partner, in a 6 hour race this year, and she did my testing and is prescribing my homeopathic remedies.
I'd say it worked out very nicely being lazy with the racks this year. Naturally, your mileage may vary.
Hugs and butterflies,
~T~