My Ficus is 25 years old. I never put it out in summer - its just too big. I have to repot on a tarp in the middle of the floor. The "hate being moved" part is a myth. Actually, they are quick to tell you when they are unhappy. Usually that goes with a move. If you keep it outside all summer than bring it in to lesser light, it will drop enough leaves to make up for the lesser light. If you change watering or over or underwater, it will drop leaves.
Mine has moved houses (and states) 4 times and has never lost a leaf. Aaronrodg12, maybe your tree loses leaves because of lesser indoor light? The sunniest window in your house is about equal to the light in outdoor shade because of window glass.