I suggest you get on your bike in plain old street clothes (with helmet though) and just go ride to have a cappuchino somewhere pleasant. Then ride home.
Think about making your bike a natural part of your everyday lifestyle instead of having biking be a big competition where every ride requires complicated preparations, special gear, and unpleasant emotions for you.
Sounds like you need to re-connect with your inner bicycle child.