As long as you don't pig out on your rest days, you should be fine. In general training regiments recommend lowering calorie intake on days you're not doing a whole lot. I try to eat more fruits/veggies on my off days and keep my calorie intake low (around 1500 calories).