I say go with them. Four is a good small group, and as long as your friend will stay with you, you'll be fine. I was scared of riding in traffic until I went on a group ride. I learned a lot by watching the other riders ahead of me, and how they handled things like intersections. At intersections they'd stop and wait for the rest of the group and we could all cross together. After that, riding on the streets didn't seem like such a big deal. I still prefer trails, but I can usually find not-so-busy streets to ride on.
You'll have a great time, and it'll be the big step onto the streets that you need, with a group of friends!
vickie