We have Mulberry trees around the neighborhood and really the birds here are far more interested in eating the other fruits such as blueberries, serviceberries and Goumi rather than the mulberries.
None of the elderberries I have ever grown in my yard were invasive at all.
