My potting soil is not the usual (what I throw in the bin for reuse is 50-75% pumice), but I have found it quite useful a second time around for amending soil when I install succulents in the ground. The rock in there helps balance out whatever compost I might also add fresh when I refill the hole (typically about half of the mix going back in is native soil). As a way of enriching our extremely poor soil, that combination has served for a wide variety of succulents, some of whom can be a bit sensitive about drainage. As I have been installing plants piecemeal, the rate of accumulation of potting soil for reuse matches its consumption.
I do not reuse soil from sick or diseased plants, cuttings that have failed to root, or anything that might possibly be contagious. The risk of bugs or whatever seems to be mitigated by the fact that plants in the ground suffer much less from the usual suspects due to the presence of predators which are absent in a patio container garden.
The question of contamination in reused potting soil (related to the contagion of disease in the container garden) does have some connection with the presence of fungus gnats. Where they are present, which is a lot of places, they have the ability to transfer disease from one pot to another just by virtue of their life cycle. You cannot assume that diseases will stay in the same pot they arrived in, given the presence of dedicated chauffeurs.
Fungus gnats can be really annoying when you are trying to start seeds and need to keep the soil moist. The same conditions that foster their reproduction are associated with small, sensitive plants. Control is possible through strategic placement of fly paper or treatment with a systemic like imidacloprid. Our spiders do excellent work. But I would imagine reused potting soil is probably going to carry over a bunch of gnats unless you do Kathy's heat treatment or something similar. I always cook the soil I use for starting seeds (pretty simple microwave protocol) because of the gnats, among other things.