Sometimes what we do for aesthetics in gardening is not necessarily what the plant would prefer
Studies have shown that daylilies recycle nutrients extracted from dying blooms to "re-use" for want of a better term. So depending how early you do it you will be inadvertently removing nutrition - whether this is at a level significant enough to have much of an impact on the plant I don't know, and gardening is often about trade-offs for appearance in any case. We do pretty much the same thing when we cut back living foliage for appearances sake.