I think GLC is right with the "kids old enough" point as one major contributing factor here.

On the Grand Tour (a week-long tour in Quebec, with days of 100+ km, lots of hills in general, similar to Cycle Oregon) there are 2200 participants, and the average age this year is 47 years old.

Who can go on 160-mile rides if they have young children at home?

(Edited to add: But this does not change the fact that not many people in their early 20s would be clocking those sort of times. It does take many years to create an endurance athlete. And those who would would be bored out of their mind before reaching the finish line!! )