You will definitely engage both muscles, and also the gluteus, or "butt" muscle ;-). Your usage depend on how you ride/pedal also -- as you push down on the stroke you will use your quards, but as you come up on your stroke, you can basically lift up and engage your hammies. This is a good technique to try when your quads are sore or worn out from a tough climb. I used it alot ot let them rest (but of course you need clipless pedals for it). If your knees are sore, then I would have your LBS to some fit checking. Tops of knees can mean saddle height (underextension), backs or sides of knees can mean something different. So it's hard to pinpoint and a fit is always recomended. Basically your knees should not hurt on a ride, if they do get it checked out.

Consider checking out some books on cycling to read up a little on pedal stroke techniques -- there are so many things you can learn about pulling up on your stroke, how to pedal a full circle, making sure to not lift the heel too much, etc. Alternative is to also ride with someone who is a good experienced rider and have them watch your stroke and give pointers.

Best of luck on your new rides!!