You can Spray with BT, which is specific to worms, and won't harm other bugs. It won't harm people either, but it's easy to wash off.
Safer makes one called Caterpillar Killer, and there are several other brands as well. Just read instructions before using. In any event, don't wait until those little inch worms turn into huge tomato horn worms.
