You don't sound hardnosed to me - it's just difficult to cycle at someone else's pace. I can ride with people that are slightly faster than me or slightly slower than me without too much problem... But people that are way slower or way faster, I have a hard time riding with. The former because they're cutting into my biking/training time and it does affect my conditioning, and the latter because I feel horrible for slowing them down.