One possibility may be timing. When I started trying to hand pollinate mine this year, I would go out in the morning before work. Very few set, but when I started coming home for lunch around 11:00 am I started having success. There were still some that did not set, but my success rate went up tremendously.
It could also be the plants themselves. My wild H. moscheutos and H. laevis had a high rate of success, but my H. dasycalyx and H. moscheutos 'Luna Pink Swirl' only set a couple of pods.
