Do you cover the drainage hole/s with anything? If not, maybe next time you repot place a coffee filter over it and then the potting soil. Not sure if a piece of window fly screening would work also.
Is the potting medium a soilless mix or is there actual soil in it?
