Just make sure when you try on gloves in the store, that they aren't totally snug or tight. Your hands expand when you get hot/very active. If you find you cannot take off your glove easily when you are hot, that's not necessarily a good thing.

Are you completely comfortable on your bike? Any neck/back pain? Any aches and pains that persist? Your arms in their most relaxed position while you are riding on your bike, should have a tiny natural bend at the elbow when riding easier parts of route. Not ramrod stiff straight.

And your handlebars and arm positioning, should be shoulder width.

Try to see when you ride along for awhile, to deliberately relax your arms...as a test, ride like ragdoll..

I have small hands, yet for the past few years I avoid XS gloves. If the glove design allows me, I will go up next size.