I think it would put your mind at ease to have an accurate body fat calculation. I am pretty much like you in build and exercise schedule; 5' 1", 106. I wear a size 2 and look quite petite in clothes. But, I have a lot of fat on my hips and butt, and I definitely notice changes in my abs, which used to be rock hard. I don't feel really comfortable in bathing suits, shorts, or even most bike shorts, unless they are the brand i get that holds the flab in with a lot of lycra. I know this sounds like whining to a lot of people, but it's almost more of a health issue than a "looking good" issue. I have upped my weight training and started doing yoga. I noticed on the days I do yoga, with a light warm up on the bike, my body fat goes down (using those famously unreliable home scales).
I think our bodies adapt to what we do and as we age, we have to fool the body and mix up the exercise, or else what we do doesn't work.