Definitely Mulberry (Morus sp.), and probably the invasive exotic Morus alba.
While not toxic to children or pets, it WILL be a big problem for your fence. I seriously doubt it was planted by humans. It is far more likely the result of bird droppings and adventitious sprouting.
It is a good candidate for removal. Plant something you want intentionally, and have it benefit the greater living landscape around you.