I would say your mistake was to try to wipe it off. I deal with this every summer. I am successful most times but sometimes I don't notice it until I have inadvertently rubbed it in. Best to try to avoid contacting it but that is very difficult for us lily lovers.