Yeah, I doubt seriously washing will help. Do put down at least an inch of cardboard, the depth of the raised bed depends on what you need to plant. Something like a tomato needs at least the depth of soil of a 5gal bucket, and a few inches more. Your bed will drain just fine with the cardboard under it, and the plant wont get access to anything in the gravel.
PS- most herbicides/pesticides remain in the soil for at least 3 years...roundup glysophates are there for a lot longer time period.
