Good idea to remove since the beans are deadly toxic.
The whole plant is toxic and, apparently, can cause bad allergic reactions on the skin so be careful when pulling and wash up well after. Be careful not to get the sap in your eyes or in any wounds. Google it. I'm not exactly sure how dangerous exposure to the leaves, stems, and roots are, but Abrin poisoning is not something you want to risk.
