5 years in a 6 inch pot? Seldom fertilized? Watered once a week?
Yes, I do think you need to change your approach.
Water is extremely important. If you are only going to water once a week, maybe they might be better off in the ground (if your soil retains moisture). If they have to be in pots, you can grow a seedling to bloom in an 8" tall tree pot or a 1 gal pot, but only if they are well watered and adequately fed (think time release fertilizer with occasional liquid supplemental feeding).
My seedlings (most of which bloomed last summer) are growing in 8" tree pots which are packed into a plastic box (one of those under-the-bed storage boxes), which *always* has water in it. (In the picture below, the seedlings in 1 gal and #1 pots were selected seedlings recently potted up from the 8" tree pots, so I can evaluate them a second year. Some of the tree pots have larger plants, which did not bloom last season (with the ones just potted into the larger pots). Some of the tree pots were just planted earlier this week.)
Water is important, maybe even more so than fertilizer. If you are worried about mosquitoes (if you use a box like I do) then you can use mosquito dunks.
For fertilizer, use a time-release one with the proper ratios of NPK, such as this one https://www.amazon.com/dp/B0071E238A/
. (Per Dan Trimmer, the proper NPK ratio is 18:6:12, so you will probably want to use more of this product than their recommended application rate. There are 18:6:12 fertilizers available, but I couldn't find a time release one that did not also contain sewage sludge, which I will not use.) At the start of the season, also (per Dan Trimmer) add something that has iron, calcium, and magnesium.
Good luck with your seedlings!