Actually, I have heard other people say that over the years where I grew up, which was an East Coast Irish Catholic town. When they said it, it was clearly not intended to be an insult and the word "pagan" was not loaded with any negative connotation in particular. Perhaps it has a meaning or usage in some parts of the country that you are not accustomed to hearing, but that does not automatically make it derogatory.
"My predominant feeling is one of gratitude. I have loved and been loved;I have been given much and I have given something in return...Above all, I have been a sentient being, a thinking animal, on this beautiful planet, and that in itself has been an enormous privilege and an adventure." O. Sacks