Presumably that can happen since each individual seed is fertilized by a different pollen grain.
Edited to clarify that it is not the pollen grain itself that travels down to the ovary. The pollen grows a tube from the stigma to the ovule, along which the male cells travel.