I think it really depends on your own body.  I used to ride a 14 mile commute (hilly) on an empty stomach and then I'd eat when I got into work.  I never felt weak or that I was going to fall off the bike! 
 
That said, everyone is different.  If you don't think you can make it without something, I agree with some of the suggestions here to try a piece of fruit or something else easily digestable and small.
But, if what you are doing works for you, why change?  Are you not getting the results/peformance that you want by eating first?