Yeah, if wasps aren't where people will get stung, we leave them alone. I know my neighbors one year kept pointing out in horror that we had a big wasp nest far up over our garage, on the second story. I said the flight path was 20 feet off the ground, no reason to make them unwelcome.
