If the numbness is in the ring and little fingers, your ulnar nerve might be getting compressed behind the elbow (ulnar nerve entrapment). I have this all the time and bike riding definitely aggravates it. There are other areas in which the ulnar nerve might be getting compressed, but behind the elbow is the most common.

Carpal tunnel affects the index, middle, and ring fingers; the compression there is indeed in the wrist (in the carpal tunnel! ).