Lockwashers.

Or, pick up a set of the leather washers at Velo Orange. I've got some on my fenders and rack and they work well. Plus they reduce creaks and rattles.