If I were you, I'd get rid of it completely. I just hate gravel in every way...That said, it might end up being quite costly though.
If you do decide to, get in a topsoil that is of equal composition and texture of your native one (sand,silt, clay) or as close to to prevent water infiltration issues.
And please don't dig it in
If you happen to have clay or silty soil, the % of the gravel in the resulting volume mix needs to be AT LEAST half to have a positive effect on drainage. In any other case, it doesn't do anything or perhaps even worsten it.