The manufacturers say to replace helmets every three years, but then they're in the business of selling helmets, sort of like the car dealers who want you to trade in just about the time you get your old one paid down enough to be almost tolerable ;-)

Helmets are made from extruded polystyrene foam, pretty much the same stuff that those foam coffee cups are made from. It doesn't break down from exposure to sunlight, or oxygen, or much of anything else. You can dissolve it in an organic solvent, or melt it if you apply enough heat (~85 degrees C). The coffee cup you throw away today will likely be perfectly usable when they dig it out of a landfill 500 years from now, provided that you wash it up a bit.

So the foam in the helmets doesn't degrade. What I have found out is that some of the ancillary stuff, like the plastic coating on the foam might get scuffed up a bit, or more likely the headband/suspension system will wear out or get all ratty-looking.

Replace your helmet if you crash it, or you drop it in a situation where it might compress or crush the foam. Replace it if the parts start to get all ratty and worn out. Replace it if you get a better deal on a helmet that you like better. But there's not really a time limit on the foam degrading, other than when it gets crushed.