You're right, the winters here are too cold for them. A lot of folks in California grow them quite effortlessly from what I understand. A greenhouse I love to visit in Lexington KY has a huge potted rosemary plant they use as a mother plant for lots of cuttings each year, and they have freesia in the pot with it. Every winter they burst into glorious bloom and scent the greenhouse.
