Ok, obviously they have an established trail through your gardens. If you can interrupt that trail by making it inhospitable for them they will find an easier route. Usually it's better to interrupt those trails in the early spring before they get firmly established. However, that doesn't mean that you can't put a damper on things till next spring. Have they been eating your stuff this year so far? You said you've been spraying with Plantskydd and using Milorganite, is that working for you? If a deer is seriously hungry they will brave most anything for something to eat. Deer are opportunistic creatures of habit. Make it unattractive for them and that will help your situation.
