Yes, geometry and tires make the biggest difference. If you find yourself falling for an aluminum bike with carbon fiber fork be sure to confirm what tire size the bike can accommodate - many are not designed for a wider tire.
of course I love my 650 B nifty swifties