I wouldn't worry too much about the leaves themselves.
What I would worry about is the soil: the leaves fell because lychee trees cannot stand waterlogged soil. It's paramount that you improve soil drainage around it if you want it to live.
Lychees also cannot be merely left to their own devices: they need regular watering outside the rainy season and fertilizer. A lot of it: commercially grown trees are "fed" two or three times a year depending on what cultural tecniques are used.
The upside is in the US they are usually pest- and disease-free trees, so that's one less thing to worry about.
