The moss is sometimes a sign that the plant is not growing quickly and may have a sparse canopy of foliage. However yours sounds plenty vigorous. The moss is not a parasite of the tree and is just "along for the ride". You can rub it off if you like but I wouldn't worry about it. Make sure the soil is well drained in that area as crape myrtles don't like to sit in soggy soil conditions.
Thanks for the question and let us know any time you have plant questions!
Q&A Library Searching Tips