Worms will come on their own, you don't need to add them. Worms like all the organic matter, and once you mound the materials in your garden, the worms will find it.
For more information, try googling "lasagna gardening" or "sheet composting".
Pat Lanza wrote a book on the subject. According to her, no need to wait for it to finish, she plants directly into the mounds of organic material immediately.
http://ourgardengang.tripod.co...
I often fling coffee grounds directly into my yard (too lazy to walk to my compost bins). It doesn't attract bugs. Coffee grounds are, I think, more of a soil conditioner than a fertilizer.
Karen