Is the over-growing branch coming from near the ground? If so it could be growing from the root stock and not be the same lime at all. The leaves, however, do not look different so my idea might be wrong. Examine it closely. If it is a hostile take-over from the root stock, it should be cut off. How old is your lime tree? The little limes may have fallen off because the tree is not mature enough to support a full load of fruit.
