It's not necessarily illegal to grow for gardening or food production. The act of harvesting opioids is what's illegal, not the act of growing. But in some states (including my own) there are also agricultural restrictions since they can become feral. If you find a jar of poppy seeds (the food) and sprinkle them over a freshly-disturbed site like a construction area or train tracks in late winter, chances are at least a few of these will pop up there the next spring.
This picture is public domain and from a poppy field in Tasmania where they grow the crop for drug production:
These fields go on for miles, and produce only small amounts of drug. That's why somebody growing 1, 2, or 10 plants is basically harmless. You'd need to set up a whole field to be on the cops' radar. And you'd probably be growing plain white varieties. I guess it's pretty, but not what I'd call ornamental and definitely doesn't compare to the beautiful pink peony-flowered on you had.