You are allowed to do what ever you feel comfortable with- no matter what anyone else thinks! Don't ever let anyone try to make you feel bad for riding the way you want to.

Personally I carry a spare tube, a mini pump, an even smaller co2 inflator, a patch kit, a park multi-tool, cash, cards and ID, food (bars & gels) and sometimes a little spare clothing (usually just arm/leg warmers). I can usually fit everything in my seat bag and back jersey pockets and sometimes I use a bottle cage carrier if I don't think I need two bottles to drink. I used to carry more, but I've found that I don't miss the stuff I leave home now.

If you want to carry a fair amount of stuff, but don't want to use a backpack you could get a number of different bags, some need a rack and some don't.

Handlebar bags can hold a small amount of stuff and usually have a map pocket on the top, but you need to be careful you don't overload them or they affect the your bike's handling.

Trunk racks usually need a rear rack to hold them up, but will hold a bit more than handlebar bag and won't affect your handling. Rivendell sells some rear bags that mount to the seat and can be used without a rack or with minimal support - they sell a stiff loop that attaches near the seat.

Panniers can hold a lot of stuff and need a rack to mount to. They are a good choice if you need to haul a lot of stuff (like a trip to the grocery store) or if you plan to do any overnight touring.