We use an extra large litter box with lids that snaps onto the bottom. Keeps the dog from being able to knock off the lid.

To keep the nosey dog from sticking his head into the box from the opening, the opening is facing the wall. And the box is placed far enough away from the wall such that our cats have no problem negotiating to enter and exit the box. But too small for our lab.

The oversized litterbox is way too heavy for the dog to push around, move it about. So he can't reposition to get into the litter box, he can't remove the top so he has given it up for a while. He is just hoping that we will forget and face the litterbox opening for an easy access. FAT CHANCE!!

Just make it impossible for the dog but still easy for the cat.

If you need more idea, please let me know. I'm sure we can find a reasonable solution without having to buy an automated litter box, child barrier etc.

Remember cats are more nimble than a dog. So this makes it easier to find a solution.