Well, you can sure enjoy the lovely scented flowers, but as soon as they're spent and it starts to form the little button that becomes the fruit, nip it off with your fingernails.
Yes, this is so the little tree can send more energy into making leaves and roots, so next year you can let it give you some fruit.
