I tried the Tarmac. At the time, the geometry was too aggressive for me, though I think I'd love it now! How's the reach? Are you able to get your saddle in the proper position? Why is it harder to handle on the downhill? If the reach is too far for you, then you can try a smaller stem or the stem set to the higher angle. Maybe you're just not used to the aggressive style? Overall sizing ratios shouldn't differ a lot from the allez, but the bars are probably lower.

Try taking it for a couple more rides to see if it's the racing style that you have to adapt to or whether it really is too big.