In my opinion, the sooner you can do it the better. The longer you wait, the more damage you'll do to the foliage. The bigger ones are also a lot easier to move before they get all of their foliage. I'm going to start moving mine this week (if my back will allow it!).
