I'm not sure I totally understand your final question, but when I did keep track of speed, I rarely averaged out all my rides as there are, as you noted, a lot of factors. I typically did the same training rides every week, so how I did on those routes from day to day and week to week told me what I generally needed/wanted to know. It was easy enough to note any significant factors, like wind, that may have affected my speed. In other words, it wasn't all that hard to figure out whether I was trending toward getting faster or getting slower.
Now I don't worry as much about my average speed. I took my computer off my bike. Not that I don't think it's a valuable metric, but I just haven't had the time or energy to focus on it, and it does take some focus. I didn't get appreciable faster until I started really pushing myself to ride with a faster group during nearly every ride. It was fun for a time but I burned out from it.
If you do make speed a gaol, just be mindful that it could easily lead to another overuse injury. Just go easy, okay!
Live with intention. Walk to the edge. Listen hard. Practice wellness. Play with abandon. Laugh. Choose with no regret. Continue to learn. Appreciate your friends. Do what you love. Live as if this is all there is.
--Mary Anne Radmacher