I've compared a 105 triple to a 105 double and I've compared rear ultegra and 105. The double was smoother than the triple, and the ultegra rear was much smoother than the 105 rear. So I imagine ultegra (or better) up front would make you much happier with the triple.

I disliked the triple so much I went withthe double (couldn't afford the bike that upgraded all over). But like yesterday when I did about 3000ft of climbing, I really wished I had a couple extra gears. But then I wasn't a huge fan of how big the middle ring is on the triple (since I wouldn't use the small ring often on most of my rides).

That's a long way of saying yes, upgrading would probably be better. But also just go with whatever is going to be the most functional for your rides. I personally wouldn't want to keep changing out those components.