Yes, we had some bags of worm castings donated for our school garden last fall. I used almost all of them to amend a new bed that we had filled with some decidedly questionable soil, and it helped tremendously. The tomato plants were sort of limping along, until I added the castings, and then they took off and were great.
I'm not sure if you could do completely without fertilizer, using worm castings though. It's hard to know how much, and what nutrients are in them, and how long it will last. I'd put down some pelleted time-release fert as well, (and we did in our tomato bed, too) if you're planting anything that's a heavy feeder.
