Conventional raised beds sometimes recommend putting a layer of cardboard down first and then putting the bed on top of that. A barrier like the cardboard is recommended especially in urban areas where lead may be present in the soil. You can "rinse" the gravel if you feel compelled but that will wash away some of the chemicals to the surrounding area.