I don't know anything about this, but wanted to second some of the other comments.

Get a lawyer if you can afford one.

Start documenting all of this (even if this issue gets settled, it sounds like this guy is likely to be more trouble in the future). Get a notebook and start logging everything (dates, times: when he drops your son off, when he picks him up, when you offer hime more time, what his respose is, whatever)- if you do wind up back in court you'll be a lot better off if you can recite exactly what happened instead of being in a situation where it's your memory/word against his.

Good luck!