I've only rarely had occasion to "pot up" crop seedlings, but will in the future to give things away in larger containers than "plugs".
The containers I have the most of are 20 ounce soda bottles, and after that, 2 liter soda bottles. Despite all the sugar and artificial sweeteners and flavors and colors, those are probably non-toxic enough for crop seedlings.
I agree with you enough not to put crop seedlings into a washed oil jug or insecticide bottle, even if it was washed, and even if a plastic surface can be cleaned down to the milligram, level or microgram level. There's no need and little motivation.
But I don't mind using WASHED laundry detergent jugs as scoops for soilless mix or fertilizer. The intrinsic toxicity of detergent is very slight, and the amount of residue in the jug is microscopic. (A fungicide or other substantially toxic chemical would be different.)
The amount that would transfer from the jug to the mix is "microscopic squared", and the amount that managed to move from the mix to the seedling would be "microscopic cubed". The amount that would persist from the seedling to the fruit would be "microscopic to the fourth power" and at that point, it would have had to be plutonium or Agent Orange with extra dioxin for me to worry.
But I never gave much thought to sowing crop seedlings IN containers that used to be toxic. I think you're right and that COULD transfer a detectable amount of a toxin to something one ate, say a root crop or baby-leaf lettuce. And many gardeners are aiming to avoid not only overt, detectable harm, but ANY unnecessary ingestion of anything harmful.
It's a good point.