I think the time to fertilize daffodils is as the foliage is dying back. Are you letting the foliage die back naturally or do you cut them off when they start to turn brown? Allowing them to stay on helps to nourish the bulb for next year's blooms. The only other factor I can think of would be if they are in too much shade.
