Answer: I can't tell from your description what the pests might be. If they fly when you brush them or come near them, they might be whiteflies. Insecticidal soap will kill them if it is sprayed directly on them. Rather than guessing, why not collect a few of the critters in a plastic baggie and take them to your local Cooperative Extension office or Master Gardener Clinic or well staffed nursery for positive identification? Once you know what you're dealing with, you can use the appropriate control
Best wishes with your lilies!
Q&A Library Searching Tips