No, that should be plenty of organic material. Your fertilizing seems adequate, and your plants look very healthy. If that picture is extremely recent, then they appear to be waking up from dormancy quite nicely and quickly for your zone. When one of your trees did bloom that one year, was it an abnormally warm year? When do your temps usually get above and stay above 60 degrees at night? For your zone, your temperature averages for lows at night are probably still around 60 in May and going back below 60 in September. Does that sound about right? If you have chilly nights that dip down "low" periodically (below 60) other than 3 or 4 months in the year, it may not be enough time for blooming other than once every few years. Of course, I don't know exactly what your temp ranges are, because I don't know what part of Virginia you live in--but that could definitely be a factor.
