What would you do -- neighbors want to be friends, we don't?

[Irulan]

Whatever you say, just be sure you practice it a few times, so you feel comfortable saying it when the time comes.

Just keep it short and simple...."Thank you so much for your offer! Unfortunately, we just can't fit that into our schedule."
If they push, just keep repeating it..."We just won't be able to squeeze it in." etc etc etc

If you are prepared, you don't have to hide from them, or worry about getting stuck in a situation you are not comfortable with.
Good Luck!

I vote for this. It's not an excuse, it's clear, polite and gets the point across.