I would go with it. If the terrain is too bad, pop it in your pocket for protection. That really shouldn't affect the reception.

Or just carry it in your pocket, period. It doesn't HAVE to be on the bars.