Answer: First off, if you are at all concerned about the stability of the bank, you should call in professional on-site advice right away -- especially before you do any digging or planting. You should also do that if you will be changing the drainage pattern in any way. Having said that, some plantings and possibly some terracing could probably help things look better.
"Rock garden" mean different things to different people. To some gardeners it is a label for a special planting of alpine plants or dwarf conifers, for others it is a catchall for plants which like good drainage. In your case, it sounds like you would like some easy to grow plants to look nice and thrive in rocky soil. Without knowing which way your slope faces it is difficult to make specific plant recommendations. However, if it is sunny you mightgrow junipers, creeping phlox, and sun-loving herbs such as the various creeping thymes. If it is shady, you might consider a ground cover such as creeping myrtle or ajuga. As you gain confidence and experience as a gardener you will find there are many plants for a slope and rocky ground.
Before you purchase or plant anything, your first step would be to run some basic soil tests and see what soil amendments, if any, you need to add in order to help your plants do their best. Your County Extension should be able to help you with the test and interpreting the results, and may even have some general gardening information available for you. The number in Dutchess County is 677-6563.
You might also consider looking at a basic gardening guide to help you get started. One I like is Gardening for Dummies by Michael McCaskey, ISBN 1-56884-644-4. All the best with your new home and garden!
Q&A Library Searching Tips