That would be a good thing. I have been stung by them in the past and it was a trip to the hospital both times.
I would move the plant, if you think the bees are no longer a threat. I have read that gas or kerosene stays in the soil and contaminates its.