I have heard that as well, but I have never seen it being a problem. I dress my adeniums and plumeria with pea gravel, lava rock, various large decorative stones, and sometimes glass. I also have most of them in all day sun on a hot deck in Texas.
I don't know sometimes someone will have that happen because they did not acclimate them well in the spring, blamed it on the rocks, and then said story spreads like wildfire, and everyone then believes that. That's my theory anyways.
Oh, but I have heard of several people using the colored aquarium rocks and they fused together in the sun.
I will take some pictures tomorrow to show all the above examples. Personally I do not like seeing the dirt and it helps keep it from splashing out and making a mess when I water. It gives a clean finished look like mulch on a freshly planted bed. If it is a new or rooting plant I usually have a larger sized stone partially buried so that when I am in doubt I can lift it up and check moister levels. Sedums and portulaca make nice toppings also.