I usually get grit from the local coop, farm store, feed store, grain elevator, or tractor store. (Whatever you call them in your part of the country.) As areas build up with population, the availability is still there, but not as easily found. The coop nearest to me has merged with Ace Hardware. They still sell the good stuffs, but you don't find it listed as a coop in the phone book anymore....
I don't know why you want to use only non-organic materials for roses or other perennials (not alpines). That's what they like. That's what soil flora likes. That's the best way to hold nutrients. That's the easiest way to improve soil structure. The benefits go on and on. Yes, it does settle, so your initial build will include a higher grade than what you want in the end. And if you don't periodically add organic materials to you soil that naturally break down, you will end up with tired soil that roses and perennials won't like. You'll need to add fertilizers on a regular basis, and that will degrade the natural balance of soil flora, too. For plants that don't normally grow in in rock substrates, composty materials are practically cure all amendments.