I have the same problem with a $400 pair of Allen Edmonds wingtips. Absolutely infuriating.

It's my foot sliding against the smooth inner sole.