last year a colony of ants set up home under one of my terra cotta pots in the patio. It looked like the "abode" was in between two slabs of concrete.

This year it seems to have taken over the pot itself and now there are signs of tunnels all around the pot, as well as the space between the concrete slabs.

Is this something that I should be concerned about, or should I just let nature do its thing and live in harmony with them? I do see the occasional ant in my place but aside from that they haven't been a bother. I really would rather not have to do anything, but I am getting concerned about the plant inside the pot (california lilac).