How do you know them? Are they "friends" you had before and started cycling with, or "just" a bike shop group or other cycling group that you met to ride with? Unless they're friends, you don't owe anyone an explanation, but as a group ride leader myself I would want to know "why" to try to fix it if this is an organized group, but would totally understand. One "regular" we had stopped coming all of a sudden, and we missed him. I shot him an email asking how he was and did we offend him or something. Turns out his son just started soccer on Saturday mornings and can't join us then, but wants to ride on other days/times. I wouldn't have known had I not asked.
Other people we either ride too slow or too fast for, and we try to help them find a group more suited to their ability and goals. The slow pace and stops would drive me crazy. My group kind of tease me that I just go and go! We only rest on "social" or coffee rides - otherwise, I'd go the full 30+ miles without stopping (that's what signals are for!) They all know they can tell me to stop or slow down if it's hot or someone is having an "off" day, and we have a good time. In fact the last two weeks we have record ride times on two of our regular routes, even cutting a ride down from 2 1/2 hrs to 2 hours flat - everyone was proud of such an accomplishment and it made us all feel good to be that much faster after riding all summer. It's nice to have a group with similar goals and riding abilities. You deserve it!