Maybe it's not toxic to deer, or they just don't like it, so only eat it when there's nothing else. Was the chunk you left in the ground, or container? Doesn't seem like it would do well in a container, to me.
By the way, the two Echium that I got started are now in large pots. One is growing like a weed, while the other just sits there. Don't know if it's the mix or the location, but I haven't killed them yet.
