Yes, seed size/weight does matter.
This is the overall consensus of opinion of the effect of seed size/weight in plant species, taken from a published scientific study, [angiosperms are the flowering plants with seeds]
"In angiosperms, initial seed weight has a strong effect on plant success. Typically, large seeds produce large seedlings..." and "...this larger juvenile size enhances survival probability, adult size, and fecundity"
From another scientific study,
"Seed size is an important phenotypic trait commonly associated with the fitness of young seedlings exposed to environmental stress"
It is possible in perennial plant species such as daylilies, that the effects of different seed sizes may disappear with time. In annual species the effects can last years, which in those cases are generations.
