Answer: It sounds like your tree has an infestation of a gall forming insect. The structures you see are grown by the plant in response to the insect's activity. Inside are the young insects which develop into adults and then emerge from the structures. Removing the fallen leaves was a good idea.
Gall insects are seldom a major concern. They don't usually warrant spraying. Occasional outbreaks are usually brought into check by their natural enemies. The tree should releaf without any long term, detrimental effect.
It is possible that another stress such as root injury from construction or herbicide injury from lawn weed control is involved in the problem. However, based on your description, it is unlikely. If your tree is losing lots of leaves, consult an arborist. If it's just a few leaves, I wouldn't worry.
Q&A Library Searching Tips