I love budget gardening! My #1 bit of advice is to find other gardeners to give you stuff cheap or free. Kijiji or Craigslist works well. I also got some free stuff by asking a neighbour if I could have some in what looked like an unwanted patch. He said to take it all!
Since I'm in a similar zone, 5b, I have a few suggestions if you see these up for grabs.
Sun:
-Hardy geraniums, clumping flower that blocks out weeds

-Euonymus, huge family of trees and shrubs with interesting foliage. I dug up a big one and found a free little one in the dirt beside it!

-Lilac, dwarf varieties are reasonable sized but if you want something big and showy, you can get that too. Only flowers a little while but so worth it.

-Rose of Sharon, flowers late in the season when there isn't much to look at. Spreads a bit so people should have extras.

-Zebra grass, very tall grass that doesn't wander around, leaves an interesting winter skeleton with poofy things on top.
Shade:
-Lungwort, an underappreciated short plant with beautiful leaves and epic pink and purple flower clusters.

-Spotted deadnettle, ground hugging plant that spreads VERY quickly.

-Hostas. Everyone loves hostas.

-Purple bugbane or snakeroot

-Ferns, forget me nots, solomon's seal, daylilies, and of course whatever else you can get your hands on. Basic ostrich ferns should be free because they spread like mad and everyone has them. I have daylilies in the sun and shade and they're all happy, some on the sunny side have a few brown leaves though.