If the quick release is tight, see if the wheel has any side-to-side movement in the fork. If that's the case the cones are probably loose. That can cause pretty noticeable wobbliness on a descent.

Handy Sheldon Brown advice: http://www.sheldonbrown.com/cone-adjustment.html