Its possible. I think we tend to kill them with kindness, when all they really need is a little dirt, good drainage and lots of sun. My large garden is an old hillside cow pasture that is very rocky. Lately there has been too much rain and the weeds are loving it. I tend to give new plantings a dose of potting soil, peat moss and some bone meal. After that they are pretty much on their own. In the spring they get a light top dressing of triple phosphate, and every other year or so I spread la light layer of aged horse manure on them in the late fall. In general they do pretty well.
Usually if the soil is too rich you will have problems with rot or lots of green and little flowering. Maybe mix your potting soil with "plain old dirt" and see if anything changes. Good Luck!
