Answer: What a wonderful display you must be enjoying!
Your memory is pretty good! Legumes do have the ability to use nitrogen from the air. However, I don't think the fertilizing is the problem. Rather, I suspect the rose has robbed the area of soil moisture and that is what is causing the problem. (Another less likely possibility is that the soil was specially prepared for the rose and thus has some unknown "different" characteristic which could account for a variance in the sweet peas there.)
Unfortunately, annual sweet peas (Lathyrus odoratus) begin to flag as soon as the weather gets hot and night temperatures are above about 65 degrees. Plants in the driest and/or warmest or sunniest locations will falter first. To try to keep your sweet peas going as long as possible, plant "heat resistant" varieties, keep them well watered, use several inches of mulch to keep the soil temperature as cool as possible, and dead-head regularly.
Good luck with your sweet peas!
Q&A Library Searching Tips