RoseBlush1 said:
Please don't put the rocks in the bottom of your bed. You will be creating a perched water table. I learned about that from sooby.
I've just put a call out to her in this response and she can explain it to you much better than I can.
What happens is that water doesn't cross the interface between fine material above and coarser material below until the upper layer becomes saturated. So rocks and gravel at the bottom of a container or similar situation actually impede drainage rather than enhancing it. The fill needs to have the same texture throughout.
Having said that, digging out the clay can create a "bathtub" that still won't drain unless maybe going three feet down got through the clay layer. It's often better to make raised beds. If you pour water into the hole now, does it drain away well? If not, you may need to figure out a way to make it do so, I'll leave that for Rick.
Another thing to consider if you want to plant more azaleas is that you need to make sure the pH of the planting material is sufficiently acidic.