It seems deer and rabbits are more troublesome where the human population is more dense and there are fewer grazing options. I live on a farm surrounded by hay pasture where I see deer and rabbits fairly often, but they rarely bother the gardens.
The last small town I lived in was nestled between the foothills of the Appalachians in the Daniel Boone National forest. The interstate and the downtown area kind of isolated a population of deer. I would often see deer standing in lawns right down town eating landscape plants. They were desperate and would eat almost anything- one friend even had conifers nibbled. Those deer looked thin and unhealthy too, it was pitiful. I lived out away from town where the forest was more lush and green, and the deer there looked healthy and didn't bother my garden.