I'm not a medical expert but it seems entirely likely you could be allergic to the pollen. Thankfully I'm not allergic to daylily pollen but true Lily pollen( Lillium) just about kills me, I can't breathe at all, and I love them
You might try some swim goggles to try and keep the fogging down to a miniumum. They are supposed to not fog up when swimming so maybe they would work for you.
I hope you can work it out, allergies suck
It's my cats world, I'm just here to open the cans.