If you really like their friendship then just use that as your recovery ride. Go ride with the faster more challenging group to get you workouts in and then go with the slower group to give yourself some nice slow recovery miles. If that still drives you crazy, maybe do half of it with the slower group then take off to do some more intense work on your own? Just a thought.