I'd concentrate building all around functional fitness, too with weight work and core work, as well as intervals on the bike. As Owlie said, build it up slowly.
You don't need to go fast to do a century...
And, if you've had consistent issues getting out of clipless pedals, you haven't found the right ones. Of course, it's fine to ride on flat pedals, but your post makes it sound like you might want to still try, because of the advantages involved.