In my experience, changes to tire width don't have a whole lot of effect on speed and rolling resistance. Wider tires are definitely more comfortable though, due to the increase in volume.

If you like the tires that you have, and they're wearing well, I wouldn't sweat it. JMHO though....