The rocks will be a problem for both young plants as well as roots. I have had to keep a bucket for weeds and one for rocks in the past. Keep at it and eventually you'll not have a problem.
Always looking for interesting plants for pollinators and food! Bonus points for highly, and pleasantly scented plants.
"Si hortum in bibliotheca habes, nihil deerit." [“If you have a garden and a library, you have everything you need.”] -- Marcus Tullius Cicero in Ad Familiares IX, 4, to Varro. 46 BCE