Stone, I have to agree with you. I have a little over 2 acres and my backyard neighbor has about 3 1/2. His plants, especially daylilies grow amazingly well compared to mine. So I went over to investigate (and to beg for some of his divisions). It turns out that his property is close to the highway and his soil is mostly clay on top. I have loamy soil with clay almost a foot down. His clay soil retains more moisture and he does not need to fertilize or water as much as I do. I have to say that although I love my loamy soil, in the heat of the summer, I'd rather have his clay mixed soil.
By the way, I can grow rosemary, but he can't.
