Often the first try at gardening in a spot doesn't go as planned. Don't give up and you'll find it works much better the second try. Compost works wonders. Forget putting potting soil in the holes. Spread a couple of inches of compost over the soil and mix it in well. Go ahead and do this in the fall so the soil will have time to mellow over winter and be ready for spring planting.
In the meantime I suggest you contact your County Extension Office and see about having your soil tested so you can many any needed adjustments now and mix those ingredients into the soil prior to winter.
Here is your Extension Office's contact info:
102 Cloverleaf Circle
Jefferson, GA 30549
Here is also a helpful online publication:
Q&A Library Searching Tips