I think that every body needs at least one easy day per week, and that tends to increase as we get older (you didn't say how old you are). It doesn't sound like you're giving yourself that. Exercise stimulates your muscles to grow, but they actually do their growing while you rest. An "easy" day doesn't have to be sedentary, but it does have to be something that really doesn't challenge your muscles or cardiovascular system - a fun cruise on the bike, a walk, playing in the pool, a long stretching session, a gentle yoga class, etc.
Also, be aware of nutrition and sleep; either of those can set you back as well.
Speed comes from what you put behind you. - Judi Ketteler