The drupes are small, and the birds eat them as quick as they are ripe...
Most people never even see the fruit!
I don't notice the limbs being a problem... except the time I had a climbing bean on one of them.
Good Spring colour too!
Found a neat saddleback caterpillar on sassafras once... otherwise look for Spicebush Swallowtail Caterpillars.
Very desirable wildlife plant...
About the only thing that anyone might find objectionable is when they send up sprouts from the roots... and there are far worse offenders in the landscape.