Unless you put up a 10 foot fence all the way around your property they are still going to wander through. The key is to keep the moving, and not having them stop for a snack. I have deer that wander through my property a lot, but for the most part, they avoid the areas I have put off limits to them. Sounds like your plan is working, don't let them freak you out so you don't enjoy your gardens.
