I recommend using a spinosad based spray such as Bonide's Captain Jack's dead bug brew. It's a semi natural product derived from a soil bacterium. Supposedly it is even safe to use on edible plants and affects primarily caterpillars, borers, etc. To avoid any effect on bees, you should spray early evening when the bees are away and since it is ideally used in the Spring, there are no flowers to attract them in the first place. It may have an effect on worms but probably because it used as an topical application, I haven't seen any effect on the earthworms. I applied once late April and I have yet to see any borers. I'll find out for sure in a few weeks when I dig up my rhizomes to divide them!
