My biggest problem has been Hibiscus Beetles, Althaeus hibisci. The larva enter the seed just when they are starting to grow, then eat the seed from the inside. This year I bag them when the buds where starting to form and so far I have not seen any sign of damaged seeds.