Thanks for all the responses!
I'm going spray the plant with a blast of water all over!
I got rid of the sections that had the aphids the other evening. The buds are already reddish in color (and getting longer) and I figured they would be blooming this week with the temps were having.
Last year I didn't see any aphids, but then I didn't get it planted until after the Spring.