If its the two of you out for a training ride, then I wouldn't worry too much... I tend to get passed on hills by my training mates as I set my own pace according to my HR... but on downhill legs I tend to shoot away and past them and we all kind've mooch back together on the flats.

Check that your training partner isn't doing something like sprint intervals - that will mean they shoot by occasionally.

And have a chat with them when you pull alongside each other - ask "wow, that was a quick burst... were you; practicing sprints/testing your strength/etc...?"

Have fun and chill - you kind've explain why this happens anyway when you write

just kind of doing our own thing and not necessarily trying to stay together the whole time

If it bothers you when they fly by, ask what they were doing - maybe its something you could do too to improve your cycling too!