Answer: It is normal for the blossoms to drop as they age, but you should be seeing tiny fruit left in their place asssuming pollination took place and the tree is mature enough to fruit. Three to five years would not be unreasonable a wait, so it may just be a matter of maturity and perhaps you will see fruit next year.
Fruit production is potentially limited by many different factors. Blossoms can be damaged by frost or by insects, or by drought. Pollination can be limited by the weather or by lack of bees at bloom time. Often some of the young fruit will drop, too, as the tree begins to self limit the crop to conserve its resources so as to carry the remaining fruit to maturity. Trees can drop their fruit at any stage for a variety of reasons related to stress, as well. Over or underwatering, overfertilizing, frost damage to the immature fruit, pest damage or diseases can all cause losses. With the American persimmon or Diospyros virginiana, the trees are considered to be dioecious meaning male or female, so you would need a fruiting pair.
D. kaki or the Asian persimmon is generally rated hardy to zone 7 (your zip code places you in zone 6) so I am not sure if you might have one of these, although most of the more popular cultivars (although not all) do not require a pollen source in order to bear fruit so I will mention them. Persimmons are not all that difficult to grow so you might want to review the following guide -- although developed for N.Carolina it is still generally applicable. One of the more interesting aspects of persimmon is its tendency to bear in alternate years, a heavy crop followed by a light or nearly non-crop. You might be seeing the results of that combined with youth and other limiting weather factors, for example.
You might also want to work with your local county extension for some troubleshooting help. I'm sorry I can't be more specific for you.
Q&A Library Searching Tips