What Tulip said. I've got one or two jerseys that I don't think I should be wearing in public, but no one but me cares.

That said, look around for looser-fit jerseys. Foska has some "freeride" jerseys, which are cut kind of like t-shirts, as far as I can tell. I'm not sure on the sizing, though. (Their selection is limited, they're not exactly cheap to begin with, and the shipping's a bear because it's based in the UK. If you register and log in, you get a bit of a discount, though.)

The jacket thing (why are we getting snow over Thanksgiving?) gives you another layer too.