I'm no expert but I've moved a lot of plants, bushes, & trees in my 72 years and I would not move them. First reason: You can plant things of complementary colors around the JM to make it not be an elephant in the room/garden. Second reason: The Wisteria needs something to climb on. You said you had it in your arbor and that sounds like the perfect place for it to grow. The Wisteria loses its leaves in the winter and so does the JM, correct. It won't be long until winter rolls around. Just think about it for a while and move it in early spring if you can't stand it another minute. I've moved so many things that did not survive. I was heartbroken. Remember, it shocks or traumatizes the trees.

My thought was also to leave the tree and make it the central feature of a new color scheme. Easier to move smaller plants instead. Anything that you've babied along for several years might not survive being moved now anyway.

Would like a pic of the area you're thinking about just for a better reference point.

I have similar issue planted dwarf acer palmatum near a weeping elm and between the two a small flowering camilla shrub which should have been moved long ago. But I have discovered that the dwarf acer and weeping elm are both deciduous while the Japanese flowering camilla is evergreen . The flowers are huge camilla's which grow three different colors and sizes on one shrub. I manage to trim the elm up and as the acer grows out filling the area with tri-colored leaves after the camilla blooms. I have kept the camilla at a small size thru trimming discreetly every other fall for many years.

Also another shrub that is trimmed twice a year spring and fall… a beautiful tri-color kudzu…which could easily take over the whole area if left unchecked, never allowing it to bloom keeps it in check.

I have a few wisterias that are a constant trimming issue . The early spring blooms are always exceptional to see it blooms after the daffodils, as the garden slowly wakes up from its winter snooze. Planted with a purple clematis that blooms at the same time. Keeping both the clematis and wisteria in check training it to grow on a pergola away from the roof and adjoining trees has managed it for many years.

Fall pic showing different fall colors . Acer palmatum, weeping elm , camilla, kudzu.

The Below the Deck Garden is about 40 feet wide by 50 feet deep so there is no moving of other things or changing the color scheme. The Japanese Maple is a deep rusty purple color and everything else is white, pink, soft blues, soft yellows. It has got to go. I'm moving it into the Asian Garden or else into the South Border where there is a level spot. Thanks for all the help!