You really do need to remove that fruit. Really. If you grow the tree well through this summer, with proper fertilizer and water you'll have lots more fruit next year and it might mature for you, too.
If you leave these two fruits on, they'll steal growth from the tree and the roots, and still not mature most likely. That would be a double loss for you. Do you want no fruit and no growth, or no fruit and lots of growth? Seems like a no-brainer to me.
