I was an overachiever in elementary school. Gifted did not exist yet, and I wasn't very social. I hated my classmates because they hated me for being (one of) the best.
I don’t know if it would have helped me – the infrastructure just wasn’t there.

Thanks to my brains I had no struggle all through H.S. and I am happy for that. Sometimes I think it is best to let your kid perform to the level they can achieve with ease instead of pushing them. This is just a comment on "other parents", as your kid seems to have an easy time.

If your kid is happy with his classmates, that is very important as well.
He can always skip a grade later and get the chance now to develop his social skills. I would have given my talent for having more friends for sure.

The tiredness could mean he is stressed, and the change would stress him more. Yes to giving him stuff to do. Art class, learn an instrument (although maybe not -that is extra stress), some other group (maybe not - he says he feels tired). I had loads of extra activities that I enjoyed: violin, pottery, girl scouts. NO SPORTS.