Yes the deer must do a nightly check and wait for the buds to get big enough to make a nice big tasty meal. I have a couple I have never seen bloom yet. I will move them this year.
I get a double whammy. The rabbits snack when the leaves start to emerge then the deer come by and eat the buds.