I see one problem with eliminating Buddleia from your garden scheme, there is no better plant in my opinion when it comes to attracting pollinators!!
Mine attract Red Admirals, Black Swallowtails, Tiger Seallowtails, Monarchs, Painted Ladies, Fritilliaries and at least three species of skippers. I wish all of my plants did that!
Oh yes, bumblebees and honeybees love it too.
I don't mind if I have to pull some Buddleia seedlings.