Find a source of well washed clean river sand to amend that clay. Do it first, lay a layer and get a tractor to plough up the soil, should mix the sand and clay fast in one go. If the clay is potters clay then you may have to make a raised bed along the wall which will be free draining. Make a few raised beds around your trees unless the trees release a resin which inhibits plant growth under them. Maybe your problem has been clay that is why your garden is not doing well. Give a serious thought to raised beds.
Regards,
Arif.
