And in a Conan the Barbarian world, where the sorcerers tended to content themselves with the dark arts in lonely towers, the question of why the pirates are nowhere near any feasible trade route is a problem. Even functional magic doesn't have to throw things too far off -- healing magic and crop magic can translate to medicine and fertilizer.
It's the inbetween worlds. Where a magical house can contain a forest or a field, or inanimate objects can come to life, one starts to wonder about whether they would really worry about food in the same way that a more mundane-related world does. When people can shift through time, it could really be difficult to have a monarchy with succession. (Hmm. . . a single king who uses time travel to reign over millennia. . . .)
The one thing I particularly objected to in The Tough Guide to Fantasyland was that the economics and ecology it said were lacking -- were things that most people would not see much of, or consider much, in a day-to-day life. But some of it would be visible. Sometimes. And when you can open a closet to a forest, perhaps that pantry does spontaneously generate food. . . .
Can be a juggling act.