Fungi in the soil isn't really anything to get worried about. Here in FL (and everywhere else that has heat and humidity) we have every fungus (and bug) floating around in the air, so we just assume there is fungus on any soil, even indoors in winter when the windows are open. I guess in AZ you see it seldom enough that it would be a bit worrisome to you. But the spores are in the air even there, and when you get rain, they drop on the soil and germinate. You can't blame the potting soil. It probably was sterile when you got it.
I'd suggest, if you think it's something that's going to grow on your fence or siding, just take it out by hand, carefully putting it in a plastic trash bag, seal it and put it in the garbage. (don't compost it, it would probably reproduce)
