I wouldn't be worried about the one-sided growth. On the small sized fruit, Pork is right, your tree is too small yet to be bearing fruit. I know how hard this may seem but you should give the tree a year to grow a bigger root system, and more top growth before you let it make fruit.
Yes, when you buy a little citrus tree with fruit on it you should take it all off. Argh. Been there, done that. Heart wrenching but it will pay off with a better crop next year. A little plant like that making 20 limes already is amazing, but also may have set it back on growth so that actually might have been a cause for the lopsided growth. Just guessing.
