I don't think it is usually a problem, I have pots with only holes on the bottom and they drain OK. I think the water draining through perhaps brings in air from the top. Frillylilly if your pot isn't draining excess water well it could be because of the rocks in the bottom. Having coarse material at the base of a pot causes the potting mix above it to retain more water (perched water table). Having the same texture throughout the pot drains better.
