I never did.
You can tell when the seed pods are about ripe. They get really big, and one end of the football-shaped pod starts to elongate. You'll see the shape change just a little. Enclose in your hand and give a gentle squeeze. If it's ripe, it will POP, so make sure you have it enclosed in your fist. Check them. If they're dark brown-black, they're mature. If they're green, discard. It's not hard when you get the hang of it.
I already yanked and trashed all of mine. They get pretty ratty looking after a hot summer. I give them time to drop a few seeds naturally, then I toss them.
Karen