I'm conflicted, too. I don't have an argument against beyond that just because something did not "sell" early does not mean that it is not a great seed to post or use in another swap, and I often participate in several. One swap's white elephant is another swap's favorite and someone in the next group may be chomping at the bit to try that particular variety.
I think I like Rick's idea of being able to withdraw seeds that the holder thinks would be better traded elsewhere, but I also think that is adding what might be an unnecessary layer to the complexity of the swap.