Another reason to remove the spent blossoms [and seed pod] is to stop the seed production by the pod just below the blossom. It takes energy for the plant to bring that pod to maturity. There are other reasons of course. And of course I too have broken off the nearby bud.
