In all the years that I've been riding remote sections of trails such as the W&OD, the C&O, and the WMRT, I don't recall a single time that I've ever felt unsafe. The handful of folks in those areas are out there for similar reasons to yours, even if it doesn't appear that way. Perhaps I've become complacent and cavalier, but why would golf clubs make you feel nervous? There's at least one golf course in the vicinity of the W&OD.

I'll take solitude with the occasional weirdo over a busier trail any day of the week...