Lynxx,
The recommended procedure is to remove and bag all but the smallest leaves, flowers and buds before spraying. This reduces the chances of missing an infected leaf. Neem , at the proper dosage, will not damage leaves unless the temperature a hot. Spraying early in the morning or late in the afternoon won't damage leaves. The Brugs will regrow another crop of leaves without suffering much damage. Just fertilize a few days after you spray.