I think most states have the same law that pedestrians are to walk facing traffic, if there are no sidewalks available. In our area, we have quite a few joggers along the roads, so IMO, you did the right thing. So down a bit and pass the walker/jogger on the left. You may not have had to hard brake, but in the future, just give them some room and pass on the left. If they are jogging/walking with traffic (which is usually illegal) a warning of "on your left" before passing is an appreciated courtesy.

You also might look into a mirror for your bike or helmet. (No pun intended!)