Are they seriously staring at you or just passing a curious glance?

If they're really staring you could try giving them a hard stare back - a kind of 'Do-you-mind...' glare. It might make them cast their attention elsewhere.