Sure, it can be done. I rarely get to ride more than 3 days a week. Building up time in the saddle and developing the mental state to slog through the tough parts are as important as the fitness. For me, regardless of the ride length, my brain gets in the way when I'm ~80% done with the ride. I think I did my first century when I was routinely riding ~65 miles on weekends without too much stopping and feeling tired but not destroyed at the end.

Can you swap a gym day for a day on the trainer and do some interval work? That "butt time" can't be replicated in the gym, and I find that the trainer is harder on my butt than riding outside.