For a race bike? Weight would matter a great deal for me.
For a commuter bike? Not so much.

I put new wheels on my old road bike and dropped a pound off it. Sure. It's cheaper and better for my health to drop those last 10 pounds off me, first, but that bike rides like a different animal right now. So smooth. Precise. I love the new wheels and don't regret it for a minute (I do regret that my first race on them was in driving rain, however!).

Wheels are a great way to drop weight from a worthy frame. Bars, stem, etc, are icing on the cake. My $0.02.