The tour company is really the one at fault here. It's a tour, not a race, and on a tour, the group does not have to stay together. When we went on our tour to the Czech Republic there was a very fast and young group of riders and some newer riders. DH rode with me most of the time, and with the fast group a couple of times. Mostly, they were into getting to the stops to drink...
Since then, we have chosen tours with a little bit less of a difficulty rating, because we are there to see stuff, not just ride. On our tour of Andalucia in May, the daily mileage was only 30, but it was incredibly hilly (we're talking Vuelta routes) and hotter than hell out. Most of the riders were about like me, but everyone is a better descender than me. The sweep stayed behind me, so I didn't end up feeling like a dweeb.
As far as regular rides go, I am the same. I am always in the middle... my average is between 14 and 16 and maybe less on super steep rides. I am happy with that, but I had to search to find a group that rides at my speed.
2015 Trek Silque SSL
Specialized Oura
2011 Guru Praemio
Specialized Oura
2017 Specialized Ariel Sport