It is not easy to keep food in edible condition. One of the charms of grain was that you could store it. Pickling, drying, etc -- but there were major problems with food with all the techniques they had.
Refrigeration. Very limited. If you had a cave or a spring, you might keep stuff cool there. Or an ice box -- but the ice house took a fair amount of technological work itself.
Some of it's overstated. Spices were not used in the Middle Ages to hide rotten meat -- if only because the spices were much more valuable than the meat! Besides, selling rotten meat was a crime. There were punishments recorded, so it had to happen sometimes, but by the same token could not be routine.
Still, fresh fruit and veggies were unreliably sourced. And could carry disease. And were limited in season.
Not to mention that anything bought meant you had to worry about purity.
Personally, I find the high tech level of magic the easiest way to finesse it all. These kinds of details can get very ugly indeed.