I had noticed that when a plant was failing (wilting and collapsing), I would always find that the pot was full of worms (and big ones at that - one 5 gal pot had 62 worms in it that I could see). I started checking the pots of other plants that looked good and were still growing strong and, sure enough, they did not have the worm issue (one or two earthworms at best). All of the pots are the same type (the black nursery style) and I have used the same type of potting soil in all of them. I do know that the soil of the affected plants is consistently ground down to a fine silt that just pours out of the drainage holes now and I am guessing that is from the soil being constantly digested and re-digested by the worms.
I do not know if the plants (mostly daylilies but other plants were impacted, too) were failing due to root damage (the roots were consistently smaller than their healthy brethren), the rapid reduction in soil level, or by the change in the consistency of the soil but I never was able to find a solution to the problem.
The soil in our yard is teeming with earthworms and we are forever mulching it so I am a little surprised that they seem so eager to migrate to the potted plants.