My philosophy is the opposite. I like to transplant stuff as small seedlings.
Do you plan to grow them in pots forever, or in the ground? Leave them outside for winter, or bring in the house? Outside their chance of winter survival would be greater in the ground.
If you plan to grow them in the ground, I'd transplant them soon, after danger of frost. They would have all summer to sink their feet in and establish in place before winter.
Karen