I have a vey nice lychee tree. The fruits are absolutely to die for, if you can actually manage to harvest some!
They come with their limitations even here, and Sarasota used to be a commercial lychee growing area . . there are many large trees in the yards of the older homes around town. They bear in alternate years, a heavy crop, then a light to non-existent crop. The tree is relatively pest resistant, but in the last few years, the Sri Lanka weevils have done some damage to the leaves. Mostly cosmetic. I planted some landscape roses in the flower bed adjacent to the tree, and now the roses have their leaves munched-upon and the lychee is left alone. Sacrificial plants?
Birds and squirrels (not to mention the fruit rats and raccoons at night) absolutely love the fruit and although I've tried many methods the only one that works to keep the critters at bay is an expensive motion-detector sprinkler. The County will very likely fine me for watering when not allowed if I get caught using it, and the tree is outside the fence in the front yard, in full view of every passer-by. We have standing restrictions here on everything except micro-irrigation.
Misti, I'm surprised you wouldn't be able to grow them in the Houston area. You are the same zone as we are here. My tree came through two record cold winters a few years ago with no problems as well.