Why don't you just be honest? You shouldn't be tied to any riders simply b/c you have ridden with them in the past. Tell them and especially the guy who was most helpful that you have enjoyed and appreciated all the help that they have given you, but now you want to go a bit faster than the group; thus, you have decided to try some other rides.

Because you won't miss anyone in the group, why does it matter what they think once you leave the group?

Go. Ride. Faster!