I find the more I ride the less I worry about looking like an idiot.
In general, I think anything shorter than a 7cm stem effects your steering stability. I put on a 5 once, rode with it for about 1000 miles and decided the wobbly hill climbing was something I didn't want to keep. I went back to a 7 and it was fine. but everyone's different.