I'm not afraid of comfrey. I think anything in excess is likely toxic to some degree. That said, I use mine primarily in ointments and lotions (I'm not a tea drinker). I also feed it to my chickens, who love it. And, it is a bee magnet when in bloom.
