Personally, I would go with what you feel best about; if speed is your main concern, then get a bike that will allow you to go faster and still meet all of your other needs (touring, commuting, road rides?).
I know that my average speed on my steel Jamis Coda, with one pannier is at least 2-3 mph slower than on my very light carbon bike. I love my Coda as an errand bike or a slow, touring kind of ride, when I am on vacation. But, I didn't use it when I was commuting. It would have taken me at least 15-20 minutes longer on that bike. I couldn't go on the group rides I do on that bike. Yet, there are people on this board who use the Coda as their #1 bike and don't see a speed difference. Perhaps my small size has something to do with it? I am not that weak! But, I think this means it's a very individual decision.