Swamp Milkweeds are slow to mature , More like a tall field plant than a true Swamp plant ,I have two that are five or six yrs old , did not make mature seed pods until their fourth year ,
Glove and soap , or soapy water and dishwashing kitchen glove , to takes the Beetles off , I get three kinds of Beetle and Aphids , Spider mites that attack the pods ,
Taking them off by hand as often and clean dry dusting the seeds , if the seeds mature , is about the only way I know of ,
I do not get good viability from the seeds from mine either , but some have grown ,
Not a Super easy plant , more like a field weed , that does not like being messed with or that does not get invasive ,
Remembering , if I gave my self a grade at growing plants as gardening it would only be a good solid "C " this is my experience . some better , some worse ,,