Certainly it sounds as if you haul around abit more weight on bike everyday, even without buying anything for the pannier.
I don't like knobby tires, but I do have Marathon tires which have smoother thread. My handlebar real estate doesn't have much except for a bell and a mount for my no-longer-used Cateye computer.
For self, speed has only counted for meeting work commuting time...and for touring -- making sure on a long trip, to get to hotel/motel/b 'n b in time....to check out nearby eating places before dark..or book accommodation right there upon arrival. My partner has a loose style of trip planning: book lst 1-2 days..then advance reserved accommodation becomes vaguer and vaguer depending on cycling distance covered and weather during trip days.
So, are you waiting for a better time in life to do a credit card bike trip?