No matter what type of tree it is, the location by the fence isn't good. It's way too close, and sounds like it's too close to the neighbor tree too. I would remove it right away before it gets any larger. It can be dug out, or you could use Round Up on it if you don't mind using that.
We have a tree next to a block wall, and it's heaving up the block wall and splitting it. We love the tree, so now we end up having to dig out all the roots that go under the wall.
