I like to remove my dead foliage when it forms a mass of dead leaves that come away very easily when pulled on. My plants are mulched well in the fall, so they are not depending on the dead leaves for protection, and in our mild winters it would not much matter any how. So really the only reason I do it is just to keep the garden tidy looking.
