Beyond the safety issues (headphones and aerobars are dealbreakers IMO), I think you should aim to be tactful and kind about the "breakup," as these people's only other "sin" is being too slow for your current goals. Try not to burn too many bridges if you can avoid it. Your current goal is to get faster and stronger, but you never know when that might change. I spent several years trying to get faster when I first started riding. Now I'm just content to ride.

Even when I was training hard, I tried to keep a good mix. Riding hard with the same group can get old.