It sounds like you need to ask your LBS about putting together a ride that averages a slower speed... and/or... a "no drop" ride (where you wait for the slower ones and encourage them to keep at it!).

I bet if it were something advertised/posted in the shop... you would get a nice group meeting up to go on those rides.

And for the no drop ride... make it very clear at the start, that NO ONE gets dropped. So, that might mean stopping at certain points, etc... and if someone wants to forge ahead... and get dropped from the front, then that is their choice.

Also, can you ride with BOTH clubs? Why just ride with one? Pick the rides you like best... and go from there???

Best of luck in finding your niche!