I wish my camera was working so I can show what gravel can do. I have an old pair of Fulcrum 3s, which I use in the rain or just after a rain when there is a lot of debris in the road. (They have the same aero blades as my nicer wheels, but they are much cheaper version and weigh a ton.) The surfaces of the blades are chipped and scuffed, and they look a bit worse for wear. That is all cosmetic, however, as the wheels themselves are bomb-proof. I would never put on my nice wheels if I thought the roads were yucky, I'm too much of a prima donna about my gear.

And no, the noise is negligible. Deep dish rims and disk wheels are much noisier (although personally, I love the sound... Whoosh! Whoosh!)