I was under the impression, you can prune lightly anytime, but heavy pruning must wait for the dormant season. What is "too far"?
I thought they could handle being pruned to the ground, but I'm not an expert, either. I would be more considered about it being in a pot vs planted in the ground. When you prune it in the fall, you might consider transplanting it into the ground.