I would think, that it might be hard to build up slowly enough and train for a century, only riding on the weekends. There is nothing wrong with an average of 12 mph, but if you want to improve, it takes time in the saddle (tits). And that doesn't necessarily mean long rides 5 days a week.
I am a fan of not calling what I do training, but I consistently ride at least 4 days a week, sometimes more. At least 2 of those rides are short, maybe 12-18 miles. One is usually long, 40-60 miles and another 20-35. But all of my riding involves hills, which helped me build endurance and strength. When I did a century, I did nothing other than my regular riding and no ride longer than about 70 miles before the ride. It was flat, but with a brutal headwind. I finished in a respectable time of 6.5 riding hours, in terrible weather conditions for the last 15 miles.
You have to be willing to suffer a little, to improve. But, I think that if you even added one more day a week, it would get easier.