My experiences are almost the opposite of yours, Lindsey. I seldom see insect or fungal damage when my plants are outside but growing in pretty much enclosed spaces such as my greenhouses, that's when I see bad stuff going on. For example, in all the years, I don't remember seeing white flies on my plumeria. When I brought those six blooming varieties inside, it did not take long to see white flies show up. I am not even sure that they will damage the plumeria, and needless to say, their eggs had to already be present on the plumeria. I guess, with the wind and sun and perhaps some sort of natural predator, those white flies were controlled when outside.
I am convinced that if I did not run so many fans in my greenhouses, I would see a lot more insect/mold damage. I have some really neat, dual-fans running in each end of my solarium, but that air-movement really doesn't reach all those plumeria leaves/flowers well enough to discourage the white flies.