I think a seed swap microbadge might have two hands exchanging seeds, so maybe something like this....
(this is just a rough idea I threw together, by no means the actual badge I'm proposing...)
.....with the covered wagon idea in the background, as Jay suggested? Or maybe even a pilgrim exchanging seeds (corn) with a Native American might lend itself to the idea of a seed exchange....although that might be too politically/emotionally charged for some people.