I think sand would wash away and would personally advise against clay. When dry it is as hard as a rock but when saturated with water can turn into a giant slimy mess.
I think fill dirt sounds like the best plan so far, but would plant bushes with extensive rooting systems on top to hold the whole thing down. Or, another raised planter might be better yet if it will fit in that space.
