Yes, try watering less often, also provide more light and airflow to speed up the drying out of the soil. You can use sticky flypaper to trap the flying adults (put it near soil level) and that soap drench (use a product made for plants if possible) sounds good. You can also try a biological treatment (Bt is an organism that people use to control mosquito larvae and it works on fungus gnats too) or insecticide (imidacloprid into the soil is effective), but I would stick with the simplest approach (soap) first.
My baby cacti do bring on the fungus gnats sometimes, so those two words are not mutually exclusive. The key word in that sentence is "baby" because I have to water my very young seedlings much more often than adults to keep them happy. I do not have fungus gnat problems with my adult cacti, because I allow the soil to dry out all the way in between watering.
