That sounds like the way I used to feel in the first mile of jogging (which I don't do any more). My problem then was just what everyone has already said about your cycling--I didn't take the time to warm up properly but just went all out from the start. Not a good idea--that's not how bodies are designed to work.

In addition to taking time to warm up, are you spinning? When I learned about spinning instead of trying to push big gears all the time, that's when I really started to love cycling. Try to keep a relatively high pedal cadence with easy gears, at least while you're new at this, and always when you're warming up. I've read somewhere that even racers do a lot of spinning when they're in training.