I'm sensitive to the silicone grippers too. It's been discussed here before and we're not the only ones. It's definitely worse in hot weather. Latex doesn't bother me, thank goodness.

I just flip the grippers inside out (some of them are printed so the logos are right ways up that way, so they're plainly meant to be worn that way). It sometimes makes the shorts shorter than I'd like, but it works.