If it is an unused area with access to a road, some idiot might have dumped something there, though. Wouldn't hurt to walk the fence line on the other side and smell and observe if there's any sign of anything. Maybe even take a soil sample or two from the other side right next to the fence?
The soil test kit you get at HD or Lowe's will do 10 separate tests for you for $10.
I'm all about trying the simple things first. I do think you need to put down some balanced fert, and maybe give them a watering can of dissolved Epsom Salts each. About 2tbsp. to a gallon of water. It's so soluble that in places where there is a lot of rain it gets depleted from the soil really quickly. In summer here, I add it to my spray for orchids and they get some almost every day.