Thanks Scatterbrain for your post. I had some plants affected the past year no matter their color. Luckily I learned fast thanks to the valuable people on here, I started removing the suspicious buds and open them to check if there were any larvae inside, and yes, they were there. I put the buds in plastic bags and sealed them tight and let them cooking under the sun (I can't burn anything and never heard about bleach).
This year I had no problem at all. Some buds at the start of the season looked a bit strange so I picked up one here and one there to open them, but they were fine.
Still a mistery on how gall midge has come here. It appeared on the third year of my daylilies and none this year, the fourth.