I would remove all the plastic from under the rock. Plastic tends to cause the soil to compact and sour. Your plants will be happier without it. Where does that drain pipe go? I would worry if its just going into a dry well.
I live in the desert - my whole yard is sand and rocks. I move them aside to plant and don't worry about them. Here is a photo of my yard, rocks and all.
Daisy
