My daughter brought home a Kumquat seed she collected in Japan many years ago. It was a kumquat she bought from a street vendor and thought was REALLY good.
To my surprise, it sprouted immediately. But... it took 10 years to set fruit. And when it set fruit, it was not the wonderful kumquat from her memory.
So your problem (I suspect) is two-fold: Your plants are not old enough AND they may not resemble your beloved kumquat.
Seeds are wonderful in that they pull from all DNA for generations back and often introduce new combinations of genetic material that no one has thought of before.
But seeds are not what you want to grow if you want your Kumquat to be your old Kumquat. If its still alive, I suggest you take some cuttings.