I would let it lie. I have dropped friends when the differences in our lifestyles became too much. Feeling guilty may be somewhat normal, but we have the right to choose who we want to spend our time with.