I'm 22. I got into cycling this past year (started looking for a bike a year ago!).

I would not have bought a bike if I didn't have someone to ride with. In my case, it's my significant other. I'm also still a bit nervous about riding on the roads. Not so much the cars any more (I do what I can do, and that's all I can do; also, most people really don't want to hit you), just cars + bad pavement around here. I have a lower risk tolerance than BF, which he hasn't realized yet.

Time and cost are two other big factors. I don't ride as much as I'd like because I'm a full-time college student. During riding season, I was lucky if I could get out on weekends.
The big thing is that this is a very expensive sport. I got very lucky in that I got a cushy summer job and parents who were willing to give me half the cost of the bike as a birthday/Christmas present. Many people don't have that luxury. The entry-level road bike was about $800. Shorts, a helmet, water bottles+cages, and a bunch of other things later...well, that's a lot of money. I could go find a sports club on campus and try something else like fencing or archery (for example) for $15 a semester and not have to buy anything.
That's also an age group that doesn't make a lot of money, and the money gets used up on other things, not least of which is student loan debt. It's also a group that will quite likely not have health insurance (not limited to 20-25 yr-olds, obviously), so a sport like cycling that is perceived to be inherently dangerous ("you're riding on ROADS?") is less likely to be attractive.