Answer: Based on your description I suspect they are budworms. These are very difficult to control because they are protected inside the flower bud where sprays can't reach them and because they are resistant to many commonly used chemical controls.
A product containing Bacillus thuringiensis (Bt) can help as they will ingest it when they eat the sprayed plant, however it will take a few days to actually kill the caterpillar and by that time it will have eaten quite a bit.
Handpicking is often suggested, however in my experience this is of limited effectiveness as once they are large enough to see they have usually ruined the flower. However it will reduce the number of adults in the garden and can help interrupt their breeding cycle.
In addition to using a product containg Bt and handpicking, it may be helpful to understand more about this pest and try to control some of the other factors that can discourage it.
This pest attacks geranium, nicotiana and petunia the most often, so you may want to avoid planting these particular plants in that part of your garden next year.
It overwinters in the soil and is usually controlled somewhat by winter cold; to assist nature in doing this I would suggest you turn the soil in winter once or twice if there is an opportunity to do so when it is freezing cold but the soil is not too wet or frozen to be worked. If you are growing the affected flowers in containers, dispose of that soil and replace it with fresh potting mix next spring.
I wish I had more encouraging advice for you. Good luck!
Q&A Library Searching Tips