On our property, we debate whether to cut it out or not

Do you have Mulberry trees in your area?

Yes. Several varieties and relatives too. I also don't like them very much. . Though the bark doesn't quite match.

Persimmon?

The bark does look like persimmon.

if it is persimmon, there should be a single, curved, vascular bundle scar in the leaf scar.

Hmm that does seem to be a close match,

My guess is osage orange

Does it have thorns?

porkpal said: Does it have thorns?

No

Then not osage orange.

skopjecollection said: On our property, we debate whether to cut it out or not

If this was at my house I would suspect hackberry, from the looks of the branching pattern.
We really need to see this plant after it leafs out.

Not sure that a tree growing through a fence like that is a good candidate for a healthy specimen down the line...

But... some trees colonize from the roots, and one of those might have a desirable form.

Not through a fence, it is some 40ish cm(1 1/2 foot) away from it..

A suggestion:

In good light, place a branch/stem/twig with live buds in your hand and photograph it clearly. If you are will to prune off a live branch/twig/stem, then this is a much easier process because you can bring it indoors, or lay it on pavement, etc.

Take clear zoomed in pictures of the buds! Many times, that will give you an answer - or at least eliminate some of the WAGing currently engaged.

I don't think it is Maclura or Diospyros sp.