Answer: Based on your description you might be seeing fungus gnats or a similar pest. These breed in moist soil and then fly as adults. There are several things you can do to stop them. One is to cover the surface of the soil mix with a layer of fine grit or coarse sand so that the surface dries out very quickly after watering the plant. This will stop the breeding cycle so the gnats would stop within a few weeks. You could also treat the soil with an insecticidal soap formulated for that purpose, Gnatrol is one brand name. This should stop the larval stage of whatever is living in the soil so control is more immediate.
However, based on your description it is also possible that there is an infestation of a flying ant or termite or similar structural pest. Sometimes these will nest inside a wall or underground and emerge in a swarm at a certain time of the year. You might wish to consult with a local pest control service to see if they can identify the pest specifically and just make sure it is nothing that requires further attention in that way. I hope it is just the fungus gnats, they are annoying but relatively simple to control.
Q&A Library Searching Tips