I ride with all men, and struggled on my hardtail. When I went to F/S I did not notice AT ALL the weight difference. I went a lot faster from the get go because it let me tackle some things that I was simply not strong enough to do on the hardtail (I would wind up putting a foot down and pushing off).
I did notice some difference when I went tubeless.
Funny, I spend a lot of time cutting grams on my roadbike, but don't really care on the MTB.