Removing items is supposed to be okay and shouldn't remove them from the dibbed/sold/bought areas. It sounds like I have a bug here that I need to fix.
In fact, if the item was completely deleted from their list, then that might be difficult for me to get back for this swap. For future swaps, I may need to completely disable the ability to delete an item if others have dibbed it. (That wouldn't stop you from making it unavailable...)
Tomorrow is thanksgiving and I likely won't be able to do much work but I will be addressing this issue sooner rather than later.