Answer: The bees are finding moisture in the potting soil and it must be the only source around. They are creatures of habit and finds ways of broadcasting their good fortune. The best way to deal with it is to temporarily cover the openings and the top of the potting soil. They'll get the message an leave after a few "dry runs". Use plastic wrap as a barrier (removing it when you're watering and when the soil is draining after you've watered). This should encourage them to go elsewhere for water.
Best wishes with your garden!
Q&A Library Searching Tips