I think it's a matter of balance. I love having butterflies in my backyard, and have specific host plants for certain caterpillars to munch on: milkweed for the monarchs, rue and parsley to attract the swallowtails, and the passion vines to attract the gulf fritillary and the zebras. I don't care if those plants get munched on (sometimes down to the stem), because that's what they're there for, and they grow right back anyway.
I took this one today of one of my milkweed plants... it's growing back:
And I noticed that one of the monarch caterpillars ended up doing its chrysallis on the rue. This one is making its way out as I write!
I'm a nerd, so I find this pretty fascinating.
What gets me annoyed are the ones I didn't invite... like those oleander moths that are trying to mess up my desert roses.