Gardening under any tree that drops seeds will require pulling the seedlings. Oaks can create a solid carpet of seedlings. They're one of the hardest tree sprouts to pull, that I've gardened under. Chopping them off at the surface with a weeder won't stop them, the roots must be pulled out.
I've got sprouts from oak, maple, hackberry, mimosa, walnut, pecan, and cherry laurel. I've pulled at least 2,000 baby trees so far this year.
Is there enough sun for Crinums under a tree?