I could be wrong but I don't think there's consensus on that. Some say you should replace them after a few years, others say it's not necessary.

If the helmet is discolored or has spent lots of time in very hot temps, like in a hot car, then I would definitely replace it.

I usually will get a new helmet after 5 years or so if there's been no damage to warrant replacing it sooner.