Could the plant be getting too dry during the day, Pratik? Those stubs do look like the buds either fell or were broken off but I do see one bud that has turned brown as well. If you water deeply in the mornings, this might prevent bud drop.
Did you have an especially cold night recently? That could also cause the buds to drop.
Otherwise, perhaps a bird did it. But it doesn't look like insect activity to me.
