How wide are your tires? Wider tires could help.
You may want to try switching to something with a bit more tread. There are several types of tires that are lightly treaded in the center and a bit more knobby at the edges. They are designed to give you fairly smooth riding on roads and a bit more bite on less that perfect roads. My hybrid came with Kenda Kourier tires which are like that. I eventually ended up trading them for Panaracer TServ tires. These are narrower than my stock tires and not knobby, but with enough tread to give me confidence.