I see kittriana shouted at me. I was having a nice slumber, listening to the breeze among the birds chirping...
I think the Juglans nigra - Black Walnut - diagnosis is correct. Arigardens could snap off a leaf or two, crush them, and sniff to get the whiff of what Walnut unmistakeably smells like.
I would absolutely agree that the Walnut should go from such a proposed/existing flower garden. The other plants Walnut generally successfully associates with are the native species it evolved with, rather than all the introduced ornamental plants that many/most people grow in their gardens.
I also see Wisteria, Oriental Bittersweet (Celastrus orbiculatus), all those Daylilies (Hemerocallis sp.), the Daffodil clumps (Narcissus sp.), and maybe some Violets.
I'd start by eliminating the most vigorous invasive species - and seeing what may be obscured beneath them. That would be the vines, Wisteria and Bittersweet. If you want to pull/dig, so be it. I would cut them at ground level and paint the stumps/cut ends with Roundup/glyphosate, but you may not wish to use herbicide. If pulling plants in a site like this, it is often much easier to do so after good soaking rains, or after a good soaking sprinkler irrigation. Two for one, the plants you are leaving get a good drink, too - and you will break up less of their roots while removing the ones you don't want. Use a board or flat rocks to perch on when working on wet soils, to reduce compaction.
Have extra pots handy for the inevitable "what is this?" that you dig out/pull out accidentally, or for things that you want to relocate anyway. They will move easier, too, if they've been irrigated beforehand.
You might also want to have mulch or compost or wood chips handy to cover bare ground that you've freshly cleaned up, to reduce new weeds germinating there. The cardboard idea works there, too - with mulch on top if you have it.
Wow, ready for Nap Round Two now...