Anything thrown into the mix can be a problem. Because sometimes the story takes the element and runs with it, and sometimes it's just an amusing interlude. It can survive as an amusing interlude, if it's amusing enough, and short enough, and gives local color or reveals character when the story doesn't move.
But stories tend to be pretty relentless about needing to go onward. Even stories with an ambling pace have more forward motion in their events that may appear at first glance.
Sometimes the notion manages to weave itself into the plot, or burst into a parallel subplot that interweaves with the main, but not always. And it can be so very hard to determine which interludes really have to go. Especially if any given one is unobjectionable, but the story has managed to accumulate several, so some of them have to go regardless.