BTChance - was your LT of 181 tested on the bike?
If it was, you would be in a zone 3 and that is an intensive endurance zone so I wouldn't expect a true bonk type of scenario unless you haven't done enough base miles in a zone 1 to 2. If that's the case you might still be getting a minibonk.
If you tested out your LT on the run, then it may not apply to your bike and you could be working at quite a relative HR even though you feel not too badly.
The symptoms sound like a sudden drop in blood glucose that comes back as you back off. So apart from training zone issues, you might be having trouble digesting what it is that you are eating at that level of activity. To remedy that I would probably try taking in a gel at about 45 min into the ride. They are easier to digest than an energy bar and deliver about 100 cal.
Aside, not related to this particular problem:
Keep in mind that you don't want to be doing a lot of zone 3 training as you get closer to your racing season. If it's an easy ride keep it in zone 1 to 2. If it's meant to be a hard ride, go from zone 1 to 2 rest periods and push hard into zone 4 or 5 for your hard efforts.
Living life like there's no tomorrow.
http://gorgebikefitter.com/
2007 Look Dura Ace
2010 Custom Tonic cross with discs, SRAM
2012 Moots YBB 2 x 10 Shimano XTR
2014 Soma B-Side SS