It sure is scary!
yesterday I left the bike lane to pass a tandem going slowly up hill.
I had looked in my mirror behind me and saw what i thought was plenty of room (wide street, wide lanes) and like with you, the vehicle that passed me didn't
a/ notice me or b/ care to move. it's a good thing i didnt' wobble or anthing!