I'm pretty much the same size as you and I agree with Emily, 54 cm is probably too big.

I bought a 52 cm just yesterday and though the fit needs a little fine tuning I'm not having anywhere near the problems you are. At least on the maiden run yesterday.