I usually carry both - prescription glasses & sunglasses. When it starts getting dark or around twilight, having my glasses makes it a lot easier to see.
I usually have a hydration pack on, so they just go in the side pocket of that. I have a pair of safety prescription glasses that work made me at some point, so I just keep those permanently in my backpack for riding.