Well, it depends.

If you intend to do serious distance, it would be dumb as you get better power, can last longer, and prevent muscular imbalance...

But if you need some more time to get comfortable with the bike per se, go back, but not for too long...