True, having your characters see an apple in full bloom in one scene and eat strawberries in another is only going to be noticed by some readers, but those who do will fault you if you say it's been a week. (Still worse if they are in the same scene.)
On the other hand, I've found it hard to keep them in perfect order. Having married off a couple in one work, the heroine and companions are departing, despite impending winter. It requires calling back in memory to a month ago, before the only leaves were the last of the oaks', and the snow didn't follow -- and last in odd corners.
Ah, the life of a writer.