Indigoiis, I'm assuming that you are using the quick release that is provided with your trainer?

Some of the aftermarket quick releases don't fit into the trainer very well, and that could definitely cause movement.

Here in Maine, I live on mine for several months out of the year. I wouldn't say that my average speed is all that much slower, perhaps a bit...