Rosemary was one of the 1st things I planted when I bought my house 20 years ago. It couldn't have been any bigger than a gallon size pot. I was just experimenting to find out where the sun would be. Well that little rosemary is about 6'x8'. It suffered some damage a year or 2 ago when someone who allegedly knew what she was doing had a different opinion of what 'a little trim' means. (I had surgery otherwise I wouldn't have had her do any work.) By the time I saw what she was doing, she's cut a large v-shaped chunk out of it. But rosemary is very hardy down here and so far it's doing ok.
First picture was taken in April 2016
The 2nd picture was taken in October 2016. I cried.
I do think our recent freezing temps (hey! it's cold for us) might have hurt it a bit but I can't imagine removing it.
Sally, why would you want to remove yours? Anyway, I wouldn't cut it back now. I'd wait until spring, whenever that might be. You can amend the soil anytime you like. And that's the extent of my gardening knowledge.