You're not riding with 3 guys, you're riding with 3 other people. We're all human and we all have bodily functions.

You need to find out their average riding speed, and if yours is different then you need to discuss in advance how the group will handle it. For example, faster riders stop every X miles or at certain pre-determined landmarks to make sure the slower riders are okay, or the faster riders take turns hanging back with the slower rider. If the only plan they will consider is "we'll meet you at the finish every day" then you need to decide if that's okay with you.