Kevin - do keep in mind that garden soil is much heavier than potting soil/compost and may not drain as readily. It is also possible to bring in soil-borne critters and diseases. Just keep a close eye on it to make sure that the soil does drain pretty well. Prolonged excess moisture might be a problem.
