Thought this was a type of birch when it was tiny, but now we're thinking it's not. Want to make sure it's not too close to the house and/or invasive. We're in Madison, WI.

To me it looks likes a fruit tree - apple or pear, perhaps?

It also reminds me somewhat of Apple or Pear; for comparison check out photos of these varieties of P. caryana which by the way is considered invasive in some areas.
Callery Pear (Pyrus calleryana)
Bradford Pear (Pyrus calleryana 'Bradford')