The moldy ones may have molded because they were not viable. I would be inclined to start the firm ones unless you have a lot and you are concerned some pathogen might have spread to them. It takes 2-3 weeks to air dry daylily seeds to a low moisture content according to research. They don't need to be stored in a fridge although it can prolong the storage time especially in a warmer more humid climate. In paper bags/envelopes in a cool dry room is fine, that's how I've stored mine. They only go in the fridge for stratification (in damp vermiculite) a few weeks before I plan to start them.
People that store them with a relatively high internal moisture content (such as yours would be after only 2-3 days) often put a piece of kitchen paper towel in with them to soak up any condensation. How effective that is I don't know because I've never done it that way.
