Have you ever checked to see how "pure" your outdoor production plants stay? I've seen estimates ranging from less than 0.5% to over 15% cross pollination, depending on variety and aggressiveness of insect pollinators.
What you're doing indoors sounds like it would keep the strain 100% pure forever, except for selection pressure favoring the traits of whatever individuals you choose to bring indoors. And I would call that "desirable selection" rather than "genetic drift".
I assume that you don't have many or any die off indoors or refuse to fruit, which would cause an "indoors" selection pressure.
I think the idea of staggering seed harvests is great. Gene banks do that, too.
]
Thanks for the article!