This is the weird thing... the Swamp milkweed was in a wet area! I threw some freebie seed out in a drier area and a couple years later that germinated and I got some beautiful plants and plenty of seed collected this year. Now, will it come back next year? I have no idea.
The main reason I tried getting rid of the lilac is it is not native to the US. I left it alone for so long because it was a great bird holder outside the window and I got some very nice pictures.
It had strong enough branches to hold feeders, too.