If you swat at a branch or shake it, does a cloud of white fly up? that's my indication that white flies are around.
If it is white flies, just spraying some neem oil has always done the trick for me. I do it in the evening. I've never seen any actual damage from white flies though so once it gets that bad.. I'm not sure I have the best advice.
