For me, it was a matter of looking at my HR when riding normally, when exerting myself and then when I started having the erratic breathing (which happens to me around 175, but I'm in my late 20s, so yours could vary wildly depending on age, general fitness, etc). When my issues are HR related, it kinda feels like asthma, but kinda not... For me, it's less a feeling of constriction in the chest and more like my throat is constricting. It sounds weird like that, but it's the best way I can describe it.
On the diff between allergy/asthma and pulmonologist:
I've been seeing a pulmonologist, but I'm looking to change. He's just not that interested that I take my inhaler on average 4 times per hour on a ride

As my GP says, "the goal is to not need the rescue inhaler, ever". I think the downside of a pulmonologist is that they may not see many active people. I'm usually the youngest by at least 40 years when I go see him and one of the few not on oxygen. That could just be his practice, but it's not terribly encouraging.
I've seen allergy/asthma specialists in the past. The big downside, in my opinion, is the pushing of allergy shots. They work for a lot of people, but I've been there done that, didn't work. I prefer not to have to justify my decision on every office visit, but your experience could be different. But, they seem to be used to dealing with younger and more active patients. They are much better at treating the underlying allergies to allergic asthma and building a comprehensive treatment plan to cover the allergy and asthma parts of the equation. So, if you think allergies might be playing a role, that may be the way to go.