Ehh, my SO runs in the same outfit sometimes You're good!
I wonder if you could use the seam taping that they use on waterproof gear? Or...hmm...is it friction wearing it out? Let me find mine and think about this.
Yeah, I think the problem is that the seam stitching is on the outside of the pants, so the seams rub against each other, and eventually they start to fray.