That's the hard part... getting rid of the other nasty stuff.
Here, where we bought property there is pasture grass & 5 ac. so there's little chance I can do away with it all. But I'm not going to water the pasture grass. It's gotta survive on it's own or it dies! Pasture grass survives. But my biggest plan is to turn most of the pasture (meadow) areas into wildflower meadows. A little at the time.
I will have less "garden" here. The property itself is it's own garden. Of course, around the house I will have my favorites but they will be chosen very carefully & judiciously.