It looks like a unicorn caterpillar, Schizura unicornis. According to my caterpillar book it eats apple, beech, birch, blueberry, cherry, dogwood, elm, hawthorn, hickory, hop, hornbeam, maple, New Jersey tea, ninebark, oak, persimmon, poplar, rose, sweet pepperbush, willow, witch hazel and some other woody plants/trees. Caterpillars only eat what suits their species, a caterpillar won't move to just any other plant.
The reason caterpillars are damaging to plants is that they remove leaves or parts of leaves which means the plants can't photosynthsize (make their own food) as much. Deliberately removing the leaves does even more damage in that way than the caterpillar may have done.