Yes, if you have inspiration for a story that can be summed up in a one-sentence summary of the plot, you can start there. If you have a nasty antagonist who will give your protagonist real trouble, you can start there.
But you can start anywhere where you get the idea. I've started with plots -- either retelling a fairy tale or ripping off one that the original author just didn't do justice to (and may even have made backstory!). I've started with nifty plot twists or clever bits of magic. I've started with -- well, actions is better than scenes, because it can be a short portion of the scene, down to something that could be contained in a picture (especially, of course, if the original source is actually a picture). I've started with themes and abstractions (though those can be a nuisance), and I've started with titles -- and sometimes the title even fit the finished story.
All of these have their challenges in that they all have something that needs development. The fairy tale plot needs elaboration; the ripped off plot needs to have the serial numbers filed off and then elaboration. The actions need an entire setting and plot to lead up to them, and lead away -- and some are inciting incidents and some climaxes, and in particular, characters have to be more complex than would be manifested in a single moment. An abstraction or title makes a plot look detailed with the need for elaboration.
But trying to force yourself to start from one place only adds unneeded difficulty.
(Though I advise people to try to get the ideas to work together if you can. It does simplify the invention process.)