At our local emergency vet clinic, they say they have more cats die around Easter due to Easter Lily poisoning than any other time of year. They have a big sign up warning feline owners of the danger. My sister had a cat a couple of years ago that munched on some plants on her deck and died from pancreatitis. So it can happen.
Some cats also are not attracted to plant material. They've apparently done studies on this, read about it a couple years ago. Here I have plant munchers. One of my cats bit into a lily leaf, but did not take any material out. I was thankful, but maybe Lefty's theory prevails here. He knows it tastes like death and took off without ingesting any.