The old ones, mostly what I have, are not sweetly fragrant; maybe the newer ones are. But the old ones have a very distinct fragrance, yes, a lot like lavender and lilacs make us remember our grandmothers.
The ditchlilies have a very distinct scent and taste.
They should though. Ditchlilies and the lemon lilies are what started them all. Grandmothers of all daylilies.