Ants won't generally hurt your plants, Greg. They are probably nesting in the pot because it's warm and moist but up off the ground. Come winter they will 'migrate' down into the ground where it will stay warmer. They're just a nuisance.
Some types of ants will sometimes 'farm' other insects to harvest their droppings, which are sweet and sticky. So they can actually import things like aphids and whiteflies onto some plants for this purpose. But if the ants aren't getting on your plants. I don't think you have any worries unless they're bothering you.
If you do want to get rid of them, for instance if you were to move the plant indoors for the winter, a bait works the best, and targets only the ants. Some ants change their diet through the course of the seasons, eating protein bait in winter and sweet bait in warmer weather. So if you bait, and the ants don't take it away, try another type of bait until you find the right thing. They will take it back to the nest and poison the queen which destroys the colony. If you only kill the crawling workers that you see coming and going, the nest will continue to produce new ants, and sometimes will divide and form two nests, so just dusting or spraying stuff sometimes makes matters worse.
Down here our County Extension service gives classes each year specifically on Ants, Ants Ants. They're fun and very informative. But I'd guess we might have a few more bothersome types of ants here than you have up there, and they are active for more of the year, as well.
