Hi Gritsgal, and welcome. Are you referring to Iochroma with an " i " ? Yes, it is related to datura and brugmansia so it is a toxic plant. Many plants are toxic, but people still grow them. If you try to eliminate everything toxic from your garden, you will go nuts.
Milkweed, tomato plants, and many weeds you probably already have in your yard like nightshade and balsam apple are very toxic. If you have a dog or horse that samples bites of your plants, you just have to be sure your dogs and horses aren't able to eat the leaves or flowers.Keep them in a pot, in an inaccessible spot, maybe? Also rigorously train the dog/horse not to eat your garden plants! As for deer, they generally know instinctively what to eat and what not to. So do most pets, for that matter.
I have grandchildren who visit me, and I also grow brugmansias which are toxic. But I've never yet seen any of my grandchildren, or my pets for that matter, interested in eating any part of any of my plants (except the tomatoes and blueberries, that is). My brugs are pruned so high that the kids won't be able to reach them until they're old enough to know better.