Mr. Rocklobster;
About 10 years ago I had the same problem and questions you do but not this site to help and only the wisdom of my grandmother and aunt. They taught me that not everything needs to be perfect and to make due with what we have. That being said, I did the same to the gravel pit at my old house and even after hauling it over to my neighbors place it seemed bottomless. You undoubtedly have a lot of dirt mixed with your gravel
and that's a good thing. Growing tired of trying to get it all out and wanting to start growing my own veggies, I just framed my beds and moved as much gravel as I could between them and started sheet composting. I still got gravel in my beds but that turned out to be a boon. It provided aeration to the ' soil ' and gave it good drainage as well. Growing carrots, radishes and turnips was, believe it or not, easier than in the dirt area that was the previous, temporary garden site.
Hang in there and keep going. Top off your beds with leaves, salad waste and anything else you can find, even shredded newspaper. You may eventually think about tilling it and in my opinion, you should! The pea gravel does wonders if you'll let it, it'll even provide the same benefits that rock dust will.
I truly do hope this helps you and please feel free to tree-mail me if you have any questions.
All the best to you.
J.R.