Melissa, when you move or move daylilys, it can take time for them to settle in. Especially if you moved a distance - say to another part of the state or another state. You say the 1st year the deer ate them so they took a hit right there. Then the next year you moved them again when you put them in a new raised bed. They've had a lot of turmoil over the last few years. This could be the reason they don't have the fragrance & the color is off. And sometimes when you move them they just plain don't like that spot but I'm betting it's simply a matter of them having to adjust to the new place & "find their feet". It might take a few years --- 2 or 3.
I'm sure others with lots more daylily experience than I will be along to give you more advice.
