Nope, not just you.
I trim when it looks like it needs it and have yet had one get mad at me for doing it wrong. They are tougher than they look and I think many people may get intimidated by the whole pruning group thing, especially when they don't know the name of their plant.
And I never paid attention to when they bloom. I see info that says they are this group if they bloom in April, May or June etc., well, what of you are in warmer zone and things start growing early?
So I prune when I get around to it and they seem happy enough.