I just remembered that my husband -- who is a little over 5'8, maybe 5'9 -- had to swap out the stem on his Giant to a 9 cm because he was getting backaches. You guys have the exact same bike, I think -- a medium 2005 OCR C3. It's just on the verge of being too big for him so I'm not at all surprised that you feel like you need to swap out the stem.
His other bikes are 54 cm and 56 cm and they fit him fine; he had a lot of problems with the geometry of that Giant.