I think you don't have to say anything to them. A gift is willingly given, not something that should be expected. I would say nothing, give them their token gift this year and next year just send a card. You don't owe anyone an explanation.