Me. All the time.
I much prefer UnderArmour shorts. (the plain ol' compression heat gear shorts that come in 4 inch and 7 inch lengths and are $25-$28 or so) I've got several pairs, and my oldest ones are about 6 or 7 years old and still doing fine. They look a little scruffy, so I use them under other clothes.
I don't generally ride in "bike clothes" for short rides under 50 miles or so, but I'll wear my wool Ibex capris (which has a chamois) when it's raining or cold. Not because I like the chamois (I don't) but because I want the wool.
"If Americans want to live the American Dream, they should go to Denmark." - Richard Wilkinson