Yes, I would try to scrub them. Clay pots are great for several types of herbs that need good drainage. You didn't specify how big the pots were, but the really good ones, which come from Italy, are very expensive. You may not be able to totally remove the stains, but scrubbing them is recommended to remove any harmful fungi or anything left in them. Use a simple soap, a wire brush, and rinse well. They can crack and break in winter weather, but I'm thinking in CA that shouldn't be a problem.