Answer: To grow hydrangeas you will need to add ample organic matter to the native soil, organic matter would include compost, old rotted leaves, aged stable manure and bedding and so on.
You might want to run some basic soil tests and see what other amendments would be needed based on the results. Your county extension should be able to help you with the tests and interpreting the results with an eye on growing hydrangeas there.
There is a certain caution here, since there is a possibility that the gravel area may have been treated with herbicides and if there has been traffic on it, the soil may be very compacted. You will need to prepare the soil deeply and may even want to grow annuals there for a year to make sure there are no unexpected problems in that area of ground.
Q&A Library Searching Tips