On a road bike? Probably three times. One of them, I was nudged by DH and then he rode off! There have been some close calls.. i.e. after a short hike through some mud. Not a good idea.
On the mountain bike, too many times to count, and those clips are loose. I can only blame myself because i get nervous, slow down, and then tip over.
Bent the derailleur hanger a couple of times, too. My habit is to unclip the left foot first. So most falls are to the derailleur side. I would change that habit if I could.