I used to use E6000 but found that GE Silicone II is better.
I drilled holes in the pieces for the taller totems and secured them in the ground with a rod. Some of my pieces are not drilled or glued, just stacked. If they happen to fall over it is on dirt or grass so nothing gets broken.