Mold spores are in the air just waiting for a comfy cozy place to settle down and grow. They like damp air and dark places. Bright lights and a gentle breeze will do a lot to discourage the mold from growing. If it does grow, a sprinkle of cinnamon will get rid of it.
When you want to grow plants indoors, use soil especially made for containers. If you want to grow plants outdoors in containers, the garden dirt is fine (if its decent garden dirt). The dirt in your yard is harboring all sorts of stuff you don't want in your house. If you absolutely must use yard dirt in your indoor container plants, sterilize it first.