Clueless? I don't think so.
My guess is that you're in the good place of not judging or categorizing people based on appearance.
However, IMHO, now that you're aware of this, you might be well served to understand the potential danger that comes with proximity to a strung out junkey needing cash for his 'fix' and be more observant of those around you.
I enjoy the pastime of being hyper-observant. I like to observe people, their body language, their facial expressions, their gait. In an impersonal world, it's amazing what you see when you look at someone's eyes...
If you don't grow where you're planted, you'll never BLOOM - Will Rogers