I know my body has set points where it "likes" to be. Both size and weight.

When I'm active and happy I'm 145 lbs and a size 10-12.
When I'm sedentary and happy I'm 165 lbs and a 12-14.
When I'm miserable I can be up to 200 lbs and a size 20, whether I'm active or not.

I have the same pair of size 12 jeans that I've been going back and forth with for the last 15 years.

When my bod is at one of it's set points, it stays there. I won't weigh 140, no matter what I try. I won't settle at 160, it's gotta be 165.

Maybe yours has a set point it wants to stay at, and your metabolism adjusts to your activity to keep you there?