I always wait until the first frost, the foliage is severely damaged or dies back. Then I don't feel at all guilty wacking back beautiful foliage. It takes awhile for ground temperatures to drop so you have a few days to dig them up if you are not expecting severe weather.
