Thanks every one. I went out yesterday and cleared an area which I had used grounds on heavily. The worm population was unbelievable.
After all this good information I have decided that the worm situation is worth the potential sort term loss of some nitrogen. I am just going to use my coffee grounds as I want to without worrying about it.
I get about 35 wet pounds of grounds a day from a Panera Bread Bakery which they save for me in 5 gallon pails I supplied them. I pick up the grounds daily at one of their slow times.