Answer: I would make the call to "take it or leave it" based on the health of the plants and the type of plants you've grown. If you have diseased or pest-infested plants, plant debris can provide a winter haven for these unwelcome guests. If you deadhead mature flowers filled with seed, you can drop them in the garden if you want those same flowers to grow next year. Weed seedlings are fine to leave as a mulch, as long as you heap them up so they don't reroot. Mature weeds that have set seed should be removed from the garden so they don't add to the reservoir of weed seed in the soil.
If you're concerned about spreading diseases and pests in the garden, manage your compost pile so that it heats up to at least 140F to kill most pests and plant disease organisms.
Q&A Library Searching Tips