The only thing that really leaches nitrogen is raw, uncomposted woodchips. (there is a good, though long, explanation here: http://blog.arrowheadalpines.c...
I agree, adding peat moss probably isn't useful and may be detrimental. Peat is usually to improve soil structure, which involves working it in.
When you say 'fall compost' you mean deciduous leaves? That may not be ideal nutrient-density-wise, but I don't think poppies are particularly heavy feeders, so nutrients may or may not be the problem, since you are at least adding something every year. Is your soil sandy or clay? Sandy soil is more likely to be nutrient-poor and need more compost or other fertilizer.
To take a different angle, has anything changed with the location, such as overhead trees growing or being pruned back to change the amount of sun?