Anemone sylvestris Madonna is a beautiful perennial that thrives in the shade. It has white blooms in the spring and stays under 6". Though I am not sure if it will look good colorwise with a pink peony but it does not hurt to try. I love this plant - vigorous but not invasive.
As LG mentioned above, Forget Me Not is a good one but this plant acts like a biennual and reseeds freely so you can't really control where they are planted. They pop up whereever the seeds fall. I don't think this plant is invasive because you can easily pull them up if you don't want them. They don't have runners roots nor invasive rhizomes.
The above two will bloom about the same time as the TP, being early bloomers. If you want something that blooms at other times than the peony, hostas, heurcheras, and Japanese painted ferns maybe good choices as long as they are not too tall.