Are you sure he wasn't just being super cautious and didn't want to pass you? I do get some drivers that hover behind me that I can tell don't want to cross a double yellow line (because really, it's not legal) to pass me... or they're not sure of whether it's clear.

Typically I will then just kind of motion them to make sure they know it's safe to get around me and that I want them to pass me - usually they do.