More like your freehub needs serviced. You might not notice that your hub isn't spinning as smoothly when you're riding. If you get the wheel going fast (while bike is off the ground) then coast, you might hear grinding or notice that the wheel will stop sooner than normal, even though your brake isn't rubbing.
Could also be bent or stiff chain link, but you'd probably hear something when you ride.