I guess it just depends if you are susceptible to the particular toxin that is in the plant. Obviously the caterpillars are not, but the fish may be, so it probably isn't worthwhile to take the risk. Birds can also eat seeds that are highly toxic to mammals, something to do with their digestive systems being unable to metabolize the toxins.
Just read up the Wiki on the toxins in Brugs, which are alkaloid hallucinogens including scopalomine - the so-called 'truth serum'. Your rascally fish will be unable to lie to you for a while, at least, Ursula.