I had a compact frame Specialized before I bought my Terry. I'd also ridden a few Giants when I was shopping around. My inseam is a tad shorter than your's and I was between and XS and a S but probably would have gone for the XS had I bought one.

The one thing I noticed about both the Giant and the Specialized (that was vastly different from the Terry) was that the steering was way different. The Terry almost felt squirley or 'twitchy' in comparison. Once I'd gotten used to the Terry I rode my old Specialized (I sold it to a friend) and it felt horribly sluggish. I believe this is due to the larger front fork rake. It means no toe overlap, but it affects steering (which isn't a bad thing...just different).

And I don't know about the XS Giant, but my Specialized did not have room for 2 water bottle cages in the frame and it wouldn't fit on a regular bike rack (I had to buy an extension rod) due to the small size and compact geometry.