I agree w other posters. Unless you weren't advised there is a hidden covenant attached to your property or you are listed on the national historical registry or state/city/county, it's your property to do what and how you'd like.
You aren't obligated to keep providing for plant sales or donations. Was there a contract for such or just the man's goodwill and particular plant enjoyment he'd like to share and it was a cheap way for the societies to make money?
If not, be your own type of steward to the land as you see fit. You should be able to find another landscape team to help. We all have different visions. Can I ask, are you new to this town and is it small? I live in one and when I started making changes I heard about it from some cousin to the original owner in 1905 55x removed in the family tree, yet as we made changes to the house and yard they would grumble to everyone. Found old photos from another family member who stopped when we had a rummage sale (didn't know them either!) and here were hollyhocks in the same place that I had planted vs 1905 when their grandma lived here and they were so happy to see them again. Anywho, there will always be busybodies and those neighbors that tsssk. Enjoy your property and do what you see fit.
Btw the iris can be lovely and not that big a cleanup!
You have a good start with your gloves and pruners. Get yourself a garden hat, bug spray, 5 gal pails for hauling, a good hand trowel and a notebook to write down your ideas. Live w the space for a full year of seasons to see what it does and what you may enjoy or positively can't stand. Take pictures to refer back to and start planning what you want to see. See if you have any low lie areas that hold water, or dry areas that are bald. We all started where you have at some point and we all continue to make changes as sometimes you plant something and it takes over the garden, fizzles out of just fits the WHAT WAS I THINKING?!
And the people here have good ideas and lots of knowledge - welcome to you. Good luck and don't let some busybodies deter you.
Added to: I love small towns, have come from them before, but sometimes when the whole town is connected - it can be kind of hard when they feel it's their place to guide you with your domicile because so and so used to live there. Good luck!
