I don't have that trouble with my arm warmers at all. They get washed, oh, I don't know, once every 5-6 rides. I'll often pull them back up after I've pushed them down during a ride, and they stay up fine.

Both my pairs are PI, but I have a pair of wool ones and a Thermafleece pair.

I also have a pair of the DeSoto arm coolers, and they do fit kind of strangely, but they stay up just as well when they're clean as when they're dirty.