That's definitely a stem raising issue.

You don't need a gym membership to build the muscles you need for riding. Your "core" is the base for your extremities, that should be your greatest strength training effort. This does NOT mean sit ups, it means working your transverse abdominus which is the deepest layer of abdominal muscles (hence the "core")
Planks and push ups are good for this but go to http://www.about.com for more
core exercises. Also a good site for cycling info. Actually a pretty good site for almost everything.

Are you sure you don't need new brakes or a brake adjustment? I'm admittedly a speed weenie but I've never had to grip the brakes that hard.