I don't have aero bars...

Your center of gravity is very tiny, so any tiny flinch left or right can easily become exaggerated. I think using them takes a lot of practice. And I don't want to be behind someone when they're learning. The TT wanna-be's are dangerous. IMHO.

Lets just say I've had some bad rides behind someone who had aero bars, because they thought they looked cool. Took said individual from being a squirrelly rider to down right dangerous.

Most cyclists really won't shave much time by using them. But if you're serious into racing, they might help your time. Just remember that shifting gears can be a challenge, and you can't reach your brakes while in the bars.