That's the same attitude I had when I used to do an evening ride with a group that was somewhat over my head. Sounds similar to yours. I didn't think I needed to eat for a 25-30 mile ride, I just thought I couldn't keep up. I discovered it completely by accident, one day I'd had a late lunch of beans and rice around 3:00, and I rode SO much stronger that night it was unbelievable. After that, I started making sure I was fueled before the ride, and I hung with the group MUCH better.