Fit is so personal its hard to say - but my experience riding a bike that was too big and my personal fit demons too long top tubes along with too long crank arms
the consequences for me:
lack of power - hampered my ability to go very fast, when I got a new smaller bike I gained about 2mph instantly

difficulty climbing and inability to climb seated - I used to use a triple set up geared much like a mt. bike and I still had some knee pain, now I can climb anything you throw at me with my standard double no knee problems

neck and shoulder pain - this was my original reason for getting a smaller bike, I'd injured my shoulder and the bike was aggravating it, so I had a fit and found out that the bike was really way too long for me. I still have to be careful to move my head around on the bike so that I don't get stiff or sore, but I am much more comfortable now.