For cycling clothes, gloves, saddle, pedals, etc., I've had great luck with shopping online. Ebay, Sierra Trading Post (my favorite!), and Campmor are great. REI has an online outlet that's pretty good too (and shipping is free is you have it sent to your closest REI store). Also, check out the sales right here on TE. I bought my bike new from my LBS, but I did see some pretty nice used bikes on Craigslist too.

I paid $42.99 for a new Terry Butterfly saddle and $19.99 for a new pair of Pearl Izumi Attack Shorts on ebay. And I picked up some Canari jerseys from Campmor for $19.99. I basically refuse to pay full price for anything if at all possible. Even my cycling shoes...I went out and tried on a bunch of brands and sizes, then checked around online and got a pair of $95 Pearl Izumi's for $24.

You can do it on a budget! Good luck and happy shopping!