I think it depends a lot on what type of plants you are growing; some years I get time to clean the pots from my veggie seedlings,-- other years... not so much. Even so, I've never had a problem.
However, this year I grew some Madagascar vinca (Catharanthus roseus) from seed, that eventually developed a fungus, so i will be more careful in regard to what I pot that up in, in the future.