Depends. Are these late developing inflos from last season? Or newly forming inflos from this season? If from last season, sometimes they abort if they are too far into development. If they are new, it could be a pest problem or a direct sun problem. Do you spray them with water when you water your plants? I avoid doing this as well.
