Answer: Based on your description, I suspect the problem is related to the tree and the tree roots rather than the soil itself. Maples in particular are notorious for growing surface roots and for stealing moisture and nutrients from plants trying to grow beneath them. The trees also block rain from reaching the ground so the soil is extra dry, and they shade the area so sun loving plants are starved for light.
It is not effective to add soil on top of the maple roots; the roots will grow right back to the surface. And with some trees, such as oaks, adding soil over the root area can kill the tree. Trying to do extensive soil amendment by tilling will not be effective either; the tree roots will foul your tiller and you will damage so many roots that you could risk killing the tree.
Coarse sand or grit is sometimes added to a clay based soil, but only in small amounts and only in addition to copious amounts of organic matter. Remember, sand plus clay makes brick so you would never add much. In this case, since you can't dig down to the clay, I don't think the sand would help much at all. What you have described is actually a scenario where there may be a clay hardpan beneath your topsoil. This can lead to drainage problems in that the clay absorbs moisture and dries out at a different speed than the topsoil. For example, you might have situations where there is water pooled on top of the compacted clay while your topsoil is drying out. But digging or tilling sand into the top soil would not solve this. Also, you would have to lift all of the plants, do the soil preparation, and then replant.
I would suggest at this point that you consider moving your garden away from the tree. The root issue and soil issue will continue to be problematic and will only increase over time. Beneath the tree, you might use bark mulch or gravel or flagstones to create a seating area with an attractive bench, a birdbath, or other focal point. Surround that with an undemanding groundcover such as Vinca minor or perhaps the running Hosta lancifolia. Then use large containers for colorful annuals during the summer. Sit in the shade and view your new garden area from there.
Otherwise, continue adding organic matter and using organic mulch as you have been to try to keep the soil healthy. Make sure you are selecting plants that can tolerate a relatively dry soil and that are suited to a shady site rather than a sunny one. I wish I could be more encouraging but I hope this helps you trouble shoot.
Q&A Library Searching Tips