It is unhealthy to be exercising in such low temps without properly clothing yourself. There is a pretty high risk of hypothermia in the temperatures you are talking about, especially when you stop. You don't necessarily need biking specific wicking stuff, in fact, the under layers can be general wicking shirts that are less expensive. You need proper gloves, head covering, and a good solution to keep your feet warm (which tends to be different for everyone), and tights or leg warmers. You could go out in regular winter clothes (shirt, wool sweater) and pants, but you would be hot when you got going and the cotton of regular shirts would be chilling and leave you wet.
Try to find another winter activity you like until you can get the gear! Hiking, snow shoeing, x country skiing... all good aerobic activities, where the wind resistance isn't so high. I understand the having to be outside thing, but you do need to be safe. I know I sound like your mother, but I'm probably old enough to be your mother!