I think you tried to give him time by visiting him several times, even though at the urging of his ex-wife.

You could always call him or send him a card at least. This gives you distance and something for him to think instead of being forced to respond whenever you are physically there.

He is dying. But it has to be you wanting to do it, not because of guilt. Right now, you don't sound even mentally in the right shape and with right energy to deal with a terminally ill person.