I see why you're reluctant- that is a beautiful display of lilies! However most of the fragrant varieties bloom later than Asiatics, so you may enjoy extending the lily bloom season with some OTs.
"...and don't think the garden loses its ecstasy in winter. It's quiet, but the roots are down there riotous." Rumi