marialebro
The rust fungus is a way of life in our humid climate. It encourages the tree to defoliate before the storm season, thereby protecting it from breakage, but I agree that it is unsightly. However it does not hurt the tree. You do the right thing by just taking off the ugly leaves; be sure to bag them and dispose of them away from your garden since the rust spores will spread, otherwise.
The Bayer 3 in 1 or any fungicide will help somewhat.
This is the best time to prune your tree but remember any tip that you remove will not bloom again that year, so be judicious. If you want the branch to regrow, leave a 4-6 inch piece, and if you want it to not grow again, like if it is in the way of a path etc., trim clear to the trunk. There should be no shock.