Aluminum is light and cheap, but since it doesn't flex like carbon or steel (it's a property of the material), every bump will be transferred to you. Even with a carbon fork. Higher quality aluminum tubing may be less susceptible to this--DBF says that when he switched from an OCR3 (the old version of Giant's Defy) to his Cannondale R2000, the quality of the ride improved a lot. My next bike will be either carbon or steel.

Do what Indy says and get the bike checked out if it feels that wobbly on descents. That's not a frame material problem. That's either you (you're not used to going that fast), or a problem that really does need to be checked!