You didn't mention your location, but I suspect Plum curculios are causing this. They are an insect that attack plums. Don't let the ruined plums remain on the ground because the adults will emerge from them and the cycle will continue.
Spraying in the early spring is a good control against this.
