I think a lot has to do w/ leg length and body proportion.
I am 5'6" and ride a 54 cm - medium - compact (I think that is the right term) frame. One of my riding partners is 5'6'' and he rides a 48cm or 50cm - extra small - compact. He asked the LBS owner "why does SHE get a bigger bike?" out: Owner responded "look at her legs"

He wears a 28" inseam I wear a 32 - 33"

We traded stems and Voila perfect fitting bikes.