I'm using 700 x 37 (actual width 36c) because we go on a lot of gravel roads and I'm still pretty inexperienced. We won't do REALLY skinny tires because of all the gravel roads we love to bike on- we would likely get too many flats. My husband had been using 37's for years, and just last week changed to a 700 x 28c tire. He loves it. I think it's heavy enough that he won't get many flats on gravel, but he finds it more agile and faster.
My 2 cents: I'd say a 700 x 28 is plenty thin enough if your top goal is not getting the highest speed possible. It will handle potholes better than 25's, and will be less of a drastic change when going from the 35's. Going to 28's will seem WAY thinner to you than what you are on now.