Just a guess here, but the gray bark, leaf shape, leaf color and veining in your photos do look a lot like Texas Torchwood (Amyris texana), also called Chapotillo, endemic to South TX. You might Google/investigate that possibility to see if you think it's a match. I JUST bought a Torchwood from The National Butterfly Center, hoping it will go here in 8b Temple. They do have a website and might be able to ID your plant. My Torchwood arrived last week. They pruned it heavily for shipping, but this is what it looks like at present:
They flower clusters of white flowers with yellow centers and produce a dark drupe fruit March-October intermittently that are edible to birds and humans. Have you ever seen these bloom? or show any fruit?