When I started setting up all the planters out back, I noticed the wasps were all over them just like bees. So I figured they're also helping to pollinate and probably eat some other insects and I just leave them alone.
The only time I bother them is when they build their nests in a bad spot... like in the crevices of the garbage bins, one was making a honeycomb on the inside of my umbrella as I was sitting under it, inside the garage where there was a gap in the wood, and a cicada wasp kept digging tunnels right under where the garbage bins are kept. Otherwise, they can visit the flowers and anything else, just stay out of my face!