Sorry for the late response, it's been crazy at work! Excellent suggestions for companion plants! I I use many of the same.
For late winter/early spring interest, I have daffodils interspersed among the plants, it's great when they bloom together.
For later in the season, in the front of the beds I use daylilies, and the tiny lily series. I use a lot of clematis as mentioned above, like Arabella between the plants and the I put the taller clematis on the fence behind the plants. I also use the various lilies, Asiatic and Orientals between the plants, and the OT and trumpet lilies behind the plants. Further, I love the bearded irises, many of which bloom at the same time as the peonies, and I also like Columbines, my Dorothy Rose has seeded throughout the garden.
As Karen stated, Baptista are also an excellent plant to add to the back of the peony beds. I also have David Austin roses in one of the beds, that provide blooms throughout the season, when the peonies are no longer blooming.
In my next garden, I hope to combine the early, mid and late blooming plants, so that I have an a longer season on interest, and I'm also working on color echoes.
I'll share some pics when I get a moment to go through them.