I totally agree with Frank; we have bald-faced hornets and yellow jackets that build nests here that are difficult to discover until they have a large population. (Under our deck, once in a large wastebasket inside my greenhouse, where I had a bunch of garden stakes corralled.) And also the occasional "ground wasps," which might be the worst.
The best thing to do is destroy or remove any small nest as soon as you see it. If you discover a larger nest that is in a place you just cannot avoid, you should really use some good wasp and hornet spray and be done with it. They are VERY protective of their nests. I don't like using poisons in the yard and garden, and I don't like killing any insects unnecessarily, but a large population of wasps/hornets can be quite dangerous.
That said, if the nest is in an area where you don't normally go near it, and you are just seeing the wasps in the garden or whatever, they generally aren't at all aggressive and unless you are severely allergic to the stings I would just leave them be. Don't slap at them or try to shoo them away. They really are quite interesting to watch!
