I'm not a man, but I think I fit your orignal profile. My everyday wardrobe contains a lot of navy, black, grey....
before I joined a team, since I have to wear team kit now, I loved to wear wildly patterned jerseys - though I have to admit that I there are some colors that I would never wear since they make me look like the living dead.