I have gloves but I mostly forget to put them on. I have these incredible calluses ... but I have a genetic condition that means I have them anyway.

I don't know how important they are in a crash. When I crashed big, I landed on my elbows anyway - but I think most people would have taken the fall on their hands (I automatically went into my "swimming pool entry with goggles" and did this great belly flop onto the gravel, splaying my arms to "spread out the splash," which, ahem, does NOT work as well no gravel as it does in a swimming pool!! Perhaps fortunately I was at the back and nobody saw...) They'd have been pretty torn up by the gravel... I suppose. But let's see if anybody pipes up and says "gloves saved my hands!" I do know that my totally minimalist buddies who don't even wear padded shorts most of the time do have helmet, toe clips, and gloves all the time, but I k now some people need 'em or their wrists and shoulders hurt. (I thank my piano teachers for getting me in the habit of good wrist placement...)
There are "lite" gloves - you might want to go the LBS and try some different kinds.