Good that you moved it into a smaller pot and removed the drainage material. I assume the new pot has drain holes. The pot is still holding more soil than is necessary. The potential danger is that the soil around the roots will not dry out soon enough to prevent root rot. In part, that will depend on how porous the potting soil used is.
I suggest that you remove all soil on the surface that is not in immediate contact with the roots. That excess soil is preventing the soil around the roots from drying out soon enough. Allow the top half-inch of soil in the root zone to dry before adding just enough water so that it reaches that same level of dryness again in about a week. Constantly damp soil will cause the roots to rot.
For future reference, plants kept tightly potted in small pots are much less likely to be inadvertently over watered.