Even if he is a 'ladie's man' he'd just be a man to not get involved with romantically, right? Suggesting a secondary group ride hardly sounds like
'your place-or mine', especially as you started the conversational rolling of the ball. Maybe ask what the co-worker means by 'ladies man'. Dangerous or just gets around? If it's the latter you can always say no if he were to ever suggest interest. IF he did ever express interest try not to be offended if he's a gentleman and backs off asap. After all, how does one know the possibilities if the opportunity isn't put to the table?

Keep eyes open and enjoy yourself is my thought, but you know better than any of us how you feel about this situation.