Those caterpillars are most likely those of the Gulf Fritillary Butterfly. They prefer passion vine and should not bother your other plants. See the following web site for a photo to confirm their identity:
Thanks for the question. Please stop in again soon!
Q&A Library Searching Tips