If you still have problems, bring it to me. I've used 105, Ultegra, and DA front derailleurs on FSA and DA cranks. The 105 or ultegra setup on the FSA cranks with a clamp-on FD was the most finicky. The DA/DA braze-on match is amazingly awesome. But, if it's an alignment issue, I will be able to fix it. I usually have to totally re-do the FD on the FSA bike after the shop messes with it. So, I learned how to do it pretty well. If the cage needs to be moved, it can be a pain. You don't have as much range with a braze-on, but it can be moved a little bit.

However, I'm thinking that since this is your tri bike with non-indexed shifting in the front, some of it is going to be user error. Maybe you aren't used to the throw of the lever yet. Some compacts are a little more finicky about what rear cog you're in when you shift (are you any more successful shifting from the smaller end of the cassette?). Even so, it should be relatively rare to get the chain to go off to the outside if the FD is adjusted right. It's a lot more common to have drop to the inside with a compact.

You might try re-setting the limit and then tightening the cable tension if it gets a little sluggish. Adjust the FD upper limit screw a TINY bit at a time, and while you're in the little ring.