Well, starting from scratch is easy, wormwise.
I had only a small area of leaf litter where there were any worms to speak of.
I found them when I planted crocus bulbs there.
One garden bed at a time was my process.
While they ARE doing good work where they are, my priority was whatever new bed I was building.
AND since I was adding nice organic matter to that new bed, they'd have lots to eat and they'd multiply.
Later, they've all gotten spread throughout again because I've now almost completed my bed areas.
Dig under whatever leaf litter or "pile" of organic matter you found when you first got there.
Mine were along the back fence, which abuts a dirt alley abandoned by the city decades ago and now full of trees.
