Can you be allergic to silicone grippers on cycling shorts and other clothing? I thought is was supposed to be elastic (latex) allergy and silicone was fine?

My grippers cause burning red marks and little red spots on my legs as well as my "hold me in" underwear and strapless bras with silicone grippers on them.

This weekend I tried a test- took an anithistamine (nose was itchy anyway) and rolled up my grippers. The edges still managed to cause red burning lines and spots appeared. These shorts are pretty new. I guess I should get the grippers replaced. If silicone does this to me, will normal latex elastic ones will as well? Any other ideas what is causing this?