I get this pain if I descend (lots of tension and stress in my neck and upper body) soon after dropping my handlebar height without letting my body re-adjust first.

I'd consider a) looking at bar positioning and potentially raising it or b) seeing if your body doesn't adapt to the positioning

So yea, it's a "fit thing", but fit is dynamic and evolving, so it may not necessarily be a bad thing, depending.