No sooner do I decide I need to sit down and brainstorm the stones, and then sort them out into those that the noble houses use, and those that are lost in legend --
-- than the muse starts to pipe up that there are the stones that you throw about so that you can finesse some issues of the setting. Ice stones that the poor can use to keep their food unspoilt in the summer. Hmmm. Perhaps there are stronger ones that the nobles can actually use to attack, and likewise with firestones. . . do firestones warm your house, or do they cook your supper too? Is it the mark of abject poverty to have a firestone that can not even make a pot of soup bubble? Perhaps a run-of-the-mill firestone works only when carefully positioned among the hearth stones so you can't carry one on your travels.
I knew the stones went down pretty far the social scale because my heroine has some -- the two whose virtues I know are an amethyst to guard her against drunkness, and a sight stone, which she has to check that papers are not enchanted to warp her words before she starts to write, but which has other uses -- and she's not of high birth. Like all professional letter writers she cultivates speaking style like the gentry, and has a large vocabulary (peasants tend to think they're cheated if they understand three out of four words when you write up a marriage contract), but the rumors of her parents have lost their place in some war are going to be just rumors. (Even if a lord runs with them to imply things.)
Still, putting some in peasant hands would finesse a good number of world-building issues.