Unless the plant(s) look stressed in some way, I'd let it bloom. As long as you hadn't planted them just this year or last fall, they should have everything they need for yours and their happiness.
After all, most people grow lilies for the flowers...
When the debate is lost, slander becomes the tool of the losers. - Socrates