I once put a bale of peat into a garden bed that we had prepared, and for years that garden did very poorly. I have the opposite and have dry, sandy soil and thought the peat would help hold moisture. However, that garden never did well. I blamed the addition of the peat, but can't be sure. The other beds that were prepared the same way, other than the peat, thrived. The only difference in the beds was the one with the added peat.
It seemed that it took years for that bed to begin to do better. It has been 23 years since we added the peat, and that bed is now very nice. When we prepared the gardens, my husband "double dug", all the beds to a depth of 6 feet, so we could not easily remove the peat that we had mixed in.
I don't have clay soil, but I have read that coarse sand is good to help amend it, along with organic matter, such as compost.