There are lilies that I have tried to catch snatching pollen, while I have either been away or just missed the lily blooming and have realized I wanted to use it, risking if it is "dirty" or not. There are lilies I have tried to retrieve pollen early in the game, first day in bloom, that it seems to go from a pre-pollen state, after just opening up, (there is probably some term for this, I just don't know it) to the pollen being exposed to oxygen/air and becoming what I call light and fluffy useable pollen. The lilies I speak of, go from pre-pollen to no pollen able to be retrieved from the anthers.
Two lilies that have done this for me are Purple Life and Stately Red. This year I opened up the flower to retrieve the anthers and bring them inside. Both of them produced useable pollen this way, whether is is less humid conditions or cooler temperatures, I don't know, but outside, they give me nothing. Forever Susan (a Griesbach lily) does this too, though I have not retrieved anthers to bring inside to further test the theory. Lots of Greisbach tetraploid asiatics grown from seed act this way too.
I think what the person explained was the he thought the plant put energy into itself (having more chromosomes to support) and not into pollen production. It is a theory, not known fact.
Interestingly enough, I do not find this to be true for tetra OTs or trumpets. Or at least not the majority of what I have around here where the ploidy is known.
I hope that makes sense. Just found it interesting. Something to think about.