I must agree to go and form your own opinion. Up until your coworker said something it sounds as tho you had no negative vibes from this person...

if he hits on you a simple, "That's very flattering, but thank you, no." will send a clear and non-angry message to him. Even him asking you out once is no big deal if he accepts the no and lets it go... I hope it's a fun and stress free ride for you!