I think if the bike is not a good fit for you, you cannot get your legs moving well enough to do speed. I remember when I changed from a trek hybrid to a bianchi. The bianchi just wanted to go go go. The difference was very obvious.
Different frame angles can definitely slow you down. It's just physics. It doesn't have to be a heavier bike or a defective bike. It just has to do with the way it was made and the way you are made. Sounds like you need a new bike, i mean, after all you are "zoom zoom" go get you a bike that zooms.