If the neighbors are fussy, or possessive, or trouble-makers, or a little crazy, you might have bought trouble, if they notice.
I have one fussy, trouble-making crazy neighbor that is hard to get along with.
Or perhaps I'm hard to get along with, when it comes to sharing access to MY yard.
I don't want to be too pessimistic or negative, but ...
Honestly, if someone messed around with anything in MY yard without asking, I would take great offense and assume that they knew they had just fired the opening shot in a war. Maybe that's just me.
But if wind blew them to the fence before, it will probably blow them right back again.
Hopefully before they notice, or hopefully they will be more understanding than I would be.
Sorry, but I'm very possessive of every square foot in my (tiny) yard.
Maybe "territorial" is a better word. Like a mean dog!
If you asked first, I MIGHT not mind a LOT, but I've learned to not try to "share" garden space, even with a Significant Other.
Since leaves will blow back, is there any long-term solution other than the neighbors raking up all their leaves? Which I guess they don't want to do.
Would it be enough to keep their leaves from blowing THROUGH the fence? You could weave plastic Venetian Blind slats through the bottom few feet of the chain-link to block things from blowing through it. That might also hide their leaf-pile.
My attitude towards neighbors is that I would rather one left a note or made a phone call, rather than just appearing at my door unexpected. I would try to discuss it with them before doing things IN their yard again. They might well consider it trespassing, and might even be right.
Or, if they are very reasonable people, "what harm was done? Forget about it."
Having lived in New Jersey for too long, I don't assume reasonable neighbors.