If its an orange, it will turn orange with cooler weather. But most oranges ripen in the spring/early summer (mandarins tend to ripen in the dead of winter for some reason). That's pretty big to winter over. And why do you think the previous owners thought an orange tree that large was a lime tree? It seems they would have figured it out by now unless they lived in the house for a very short time themselves.
How long have you lived in this house? A full year? It takes a full year at least to figure out all the eccentricities of a new yard. One more question: What does it taste like? After tasting, do you feel like making daiquiris? (Oops, two more questions).
