I consider blueberries, Nanking cherries and some other small fruits to be ornamental, but grow them for the fruit. For example, the foliage of seaberry, as well as the orange fruit, are very attractive. Chinese dogwood, C. kousa, is very ornamental and produces an edible fruit. I have the variety "Big Apple". Maypop (P incarnata) does an edible fruit, but it is only marginally edible in my estimation. They do not usually ripen fully in this area though. A beautiful flowering vine.
I'm a big fan of the Autumn Olive. Although the fruit is not eaten by many, I do like the taste. I'm told that it also makes an excellent jelly, which I am sure is correct but have not done that. This is a very pretty small tree or large bush with silvery foliage, and is very productive. In the fall, it is covered with thousands of small, bright red fruits. Highly recommended.