I do this with arm warmers. Get the clear bathroom silicone caulk in the small tube.

I've never tried it with shorts or tops, though I've threatened to do so with my horrible team shorts that ride up.