Answer: In all honesty sun all day from nine to after six should be plenty of light for a vegetable patch, so I suspect the problem is with either the tree stealing water and nutrients (the roots can reach far beyond the tree canopy) or with the soil itself.
The best way to start improving soil is run some basic soil tests so you know exactly what amendments and nutrients are needed. In addition, garden soil is almost always improved by adding copious amounts of organic matter such as compost, rotted leaves, old stable manure and bedding, etc. In some extreme cases it is a better idea to try raised beds than to dig down and improve the existing soil (these can even help with a slope and are great for limited space gardening).
Your County Extension (696-3500) should be able to help you with the soil tests and interpreting the results.
Q&A Library Searching Tips