Calipers matter. In my experience trying to stop from slow speeds (anticipating the stop) on a steep downhill grade (almost always catch a red light at the bottom), Ultegra calipers are a huge step above 105. I will probably upgrade my 105 calipers to Ultegra or Ultegra-SL before I upgrade my shifters because they bother me so much. I can lock up a wheel without too much effort on the flat by grabbing hard, but there's something about these calipers where some added gravity is just too much for them.

I know that some other manufacturers (SRAM, Campy), tend to have similar functionality in lower end models as higher end for all components, and weight and design are the primary advances made with more $$, but perhaps not with everything, like your calipers.