HI Kat-
I have a cross bike as well and often search out the gravel roads to ride-
there's lots of pro's to riding the gravel roads- I usually find there is much better scenery and more hills, way less traffic, and because I find the speed to be a bit slower than riding on paved roads, I think I get more of a strength work out for my legs.
Things to know-be careful if there are areas with lots of thick gravel- your tires will start to slide/skid if you are going fast or turning- you can control it, and just like accelarating in a car-pedaling adds traction so it's better than coasting.
Also be wary of cars- on a paved street there's more of them, but on dirt roads there's more dust kicked up by passing cars- so I highly recommend wearing glasses if you don't already, or even goggles if you are sensitive to the dust.
You'll also get dirtier riding on the dirt roads- but I think that's fun 
Folks I know around MI think the fall is the best for riding on the dirt roads- less bugs, less mud, getting ready for cyclo cross season - which I haven't tried yet, but am hoping to learn more about it this year.
Good luck and enjoy your rides!
"the greatest that can be.... One Little Victory"