Very good question John. I just dug up three daylilies today, but did not see any earthworms.
On another note, I was digging around the flagpole a few weeks ago and up with a shovel full of dirt came the biggest worm I have ever seen! It was about a foot long and as big around as my pinkie! I wanted to get a picture of it, but it got away too fast.