Nope, I totally get where you are coming from! A year ago, I might have felt like you were jumping all over my nerves on this one, but I've definitely mellowed.
Generally, veggies have more fiber than whole grains too. I think the idea is that there is nothing in whole grains that you cannot get from vegetables, so we don't NEED them. Even the starches (which is the part that is utilized for fueling workouts) can be gotten from other types of plant matter (like potatoes or root veggies or fruits). I fuel best with these anyway, and they don't ever cause me digestive distress like some of the grains do.
I saw a comparison in a blog somewhere (I'll see if I can find it) where the person compared the soluble fiber content of oatmeal (said to improve cholesterol) with the fiber content of spinach (I think it was spinach!) and how it would work way better than oats but no one would buy eating a bowl of spinach for breakfast. Plus, oats have lobbyists!
Granted, there are a lot of things to be said for the taste, social aspect, and enjoyment of eating grains and most paleo proponents accept this as a fact and expect that no one will be 100% strict 100% of the time.