I can definitely empathize with you about lacking the energy to deal with the situation. I'm feeling the same way about my father, with whom I have a good relationship. I can't imagine who hard it would be if I were feeling the way about him that you're feeling toward your father.

I also think that you shouldn't go and that you shouldn't feel guilty about it. You've said that your father didn't want to have a relationship with you and basically hardly acknowledged your existence, which means he made his choice long ago.