Is there some kind of trick to growing these? I let them self sow (the colors I like) and always have some growing, but it isn't long before they get all brown and dead looking. Is this mildew? How can I avoid this, or should I just cut them back. Is mildew contageous?

I do not know much about selfsown gloriosa, but I have had some mildew experience. A 10-20% milk water solution sprayed on works well to keep mildew in check.
Worth a try
Maybe better that than chopping them down.