Well, this sounds horrible. I don't have kids and I have no idea how hard this must be on you.
I wonder though... for him to challenge the visitation agreement... won't he need a lawyer? Lawyers don't come cheap and you mentioned something about him not having gas money to drive 160 miles?
Will he even have the money to pay for a lawyer?
And like Knotted said... I think he's trying to get an extended visitation with him so he can skimp on child support... and yes, even get it from you. Sounds like he's doing what he can to get money.
"Life is not a journey to the grave with the intention of arriving safely in a pretty and well preserved body, but rather, to skid in broadside thoroughly used-up, totally worn out, and loudly proclaiming: WOW WHAT A RIDE!!!!"