I have surprised myself often enough that I stopped (on purpose) listening to that voice, but I know it well! (It's the same one that said "oh, they were being nice to you" after the ride or after any comments about my strength... okay, and the same one that now says "well, you were fast for a year or two, but that's over now...") I taught at a school for bright students with learning disabiltiies, and many of them would tell me how stupid they were... until I realized how hard it was for me to truly believe I was a strong rider, despite all kinds of in-my-face evidence, I had always wondered if they were playing head games with me... now I *know* better.
But... I still maintain that it doesn't need to hurt. I didn't think I could make it because of my body's signals, and I'm sure I could push beyond those signals... but I've found I can increase my fitness an awful lot without doing that, and if that level of push makes me 2% better, the 50% increase in risk of injury that pushing that hard gives me ain't worth it. I have strong legs, but not a gifted athlete's response time.

Knowing when it's my body's voice and when it's my history's voice... that's the trick ;=)