Alli, I don't think the Perlite could possibly be a contributing factor to the moldy issue. It's completely inorganic, like light lava rocks. Mold needs moisture and organic stuff to grow on.
from Wikipedia
"Perlite is an amorphous volcanic glass that has a relatively high water content, typically formed by the hydration of obsidian. It occurs naturally and has the unusual property of greatly expanding when heated sufficiently."
