I'm no expert on fruit trees but in my limited experience if it's not rabbits it's voles and deer. We've lost a couple of trees that were too big for the white spiral tree wraps from bucks rubbing their antlers on the trunk down to heartwood. An apple tree that had been planted several years was munched on by voles. So I would keep wrapping as long as practical because the damage often cannot be undone.
You could leave one or two unwrapped and see what happens, but the damage can vary from year to year leading to a false sense of security. If you put something like metal mesh too small for voles/mice around the trunks with plenty of room for tree growth then you could leave it on and wouldn't need to re-wrap for a few years until the tree got bigger.
Cross-posted with Greene but we seem to be on the same page.
