Also, don't forget that a horse can see behind as well as in front. Eyes on the side of the head confer that advantage.
Now, I have never claimed to know exactly what is going on in a horse's mind, but I would guess that a peleton would look a lot like a pack of something carnivorous coming up far too fast. To a horse, that means it is Get Out Of Here Time. Better to be safe than dinner, and all that.
Neither are horses the smartest creatures God made.
Oh, and one other thing, paved roads don't have good traction for steel-shod hooves.
The Rule, if there is one, is to be very respectful of a horse's space. To do otherwise is to invite a real wreck.
IMIHO (In My Insufficiently Humble Opinion)
Give big space to the festive dog that make sport in the roadway. Avoid entanglement with your wheel spoke.
(Sign in Japan)
1978 Raleigh Gran Prix
2003 EZ Sport AX