It might help your comfort level if you got a mirror... Not that that excuses the inconsiderate cyclists - I don't like people passing close by without telling me either, but at least with a mirror I know they are coming.