I am on the larger side of the bike rider/racer spectrum (5'9", 170, size 10), and I can assure you that it is possible for you to stay with (or drop) those skinny chicks you recruited! For a double, the key for me is a sustained endurance pace - and I don't think size (within reason) is a disadvantage there either. You're pretty darned close to your summer weight right now, even if you think that was not enough.

When is your double?

I am also trying to get a few more pounds off. I am at my last year summer weight right now, and working it slowly down. I am a Weight watchers lifetime member from whem I lost 60 lbs. with them about 20 years ago in college (yikes I am old!). After having a kid and dealing with a thyroid auto immune disorder, it makes the weight a challenge.

Hang in there - you can do it!