"I'm sorry, but" doesn't bother me at all, and if it did, I would probably be considering the context of the statement that followed. Only blatant grammatical speaking errors bug me.
Bless his heart, well... my ex-mil and family used this constantly. They were from Texas and I knew it was common, but it bugged me. It felt religious to me. I know, I am putting my own issues on a regional saying, but I could never warm to it.