I sow seed in 3.5" pots, usually 12 or less to a pot, and they can get pretty big and crowded before lining out, to the point where the pots are bulging and I have to keep them in saucers, constantly wet. When I'm separating them, even if I have 30 from the same cross, I'll pick the 10 most robust of the bunch to line out. Usually there's a pretty distinct point of size and vigor drop-off. This depends on how much room you have and how many seedlings you can reasonably care for. If I had summer rainfall and a big field, I'd plant more.
I've seen a lot of very good things come from difficult, 1-seed crosses from powerful parents, particularly tet. conversions, so I think 10 seedlings is enough. Even 5 should give you a pretty good idea of the potential. If I have 30 seedlings from a cross I have a good feeling about, I'll still discard the runts, but hold the strongest of the remaining seedlings in the shade in a 1-gallon pot until the lined-out sibs bloom. Then, depending on how the ones in the field looked, I'll discard them or line them out for the next season. I don't know about selecting based on seed size, I'm not sure that's an indicator of plant size or vigor.
In this climate, I've had the best results hybridizing with plants that do well for me. They're not always the fanciest flowers, but in the end, I'm looking for performance; mostly along the lines of substance and consistent opening ability. An old maxim that I try to follow is "never cross two flowers with the same fault". I'll still use new, first-year, "unproven here" plants, and hybridize by their catalog picture, but optimally, the plant should show me something, or possess a compelling trait which I want to see in the offspring. Reports from other growers and 'real-world' pictures from the database here help a lot in that case.
When using new introductions there's always faith involved. This spring I received a small single fan of 'Heavenly Pink Lady' from Gossard. It put up a 10" scape with 3 buds, but the flower was so nice I used it without prejudice. That little thing even set two pods from Victoria Josephine which should ripen in a couple of weeks. I've seen some very nice Vicky Jo kids, and those are some of my most anticipated seeds. It's also fun to plant some wild & weird crosses, I probably do more of those than I should.
Most seedlings are going to resemble one parent or the other, but that shouldn't be seen as a problem. Daylily hybridizing is a long-term process. Occasionally you'll get that 'lightning-strike' from two named cultivars, but incremental results are how you build a solid program and flowers with a unique look.
No matter how much you analyze and plan, it still boils down to luck, but, just as with most things, 'luck' can be steered. Many years ago Bryant Millikan told me that it's not so much hybridization skills that make a good hybridizer, it's seedling selection. I've seen this play out in the programs of several hybridizers. If you want better branching in your lines, select plants and seedlings which have good branching. Sometimes those choices are hard, but flower attractiveness seems to be more easily manipulated than plant vigor and habit. EMO and substance are, by geographical necessity, "my thing", so I try to prioritize those traits. In the 90s, Pat Stamile decided that he didn't like the way most daylilies looked in the morning, so he selected based on opening ability, and in a couple of generations many of his flowers were nocturnal-extended or EMO, while still having wide, round form, substance, ruffling and other contemporary traits. The seedlings I've grown from those plants tend to follow suit. Moldovan's plants were always known for color and branching. If you're able to visit a hybridizer's garden to buy seedlings, pay attention to pedigree and the characteristics which are important to you, because everyone sees daylilies from a different perspective. I've obtained some real gems that way.