I have a ginkgo tree in the middle of my stacked stone herb bed in the middle of the garden. It still has most of its leaves and we have had a few hard frosts / freezes down into the mid 20's here. It has been there about 15 years , I brought it home from an herb conference and it was about 6'' with 4 leaves. Guaranteed to be male.
The photo above is of the fruit and somewhere I have a few seeds I need to photograph.
The reason you do not see many trees with fruit is because they are outlawed in several places , and you are suppose to be able to sell only the male trees. The fruit of the ginkgo smells like strong puke when it is ripe.
I think this is an interesting tree, love the shape of the leaves, and because the branches are few and it grows slow can be used in the garden with out risk of shading out other plants.