Sometimes you can remember what the original seed crystal was. Sometimes as precise as thinking I was reading this here book, and on this here page, this thing happened, or was described, and my muse went, What a waste, using that as a throw-away, or as backstory, or totally ruining it by having this happen instead of that. And then developing onward from there.
And sometimes you maybe remember what the original idea was, but as soon as you sat to write it, it must have plopped in a supersaturated solution that was just waiting for a seed crystal, and it all formed, as if instantaneously, and you can't remember what parts came first and which after. (Or maybe it just feels that way and it did grow.)
And sometimes you look at a complete and intact structure and no clue how it started, how it fit together, and what was needed to make it all work.
Fortunately, this is one of the mysteries of writing that needs only philosophical contemplation if you feel like it. You don't have to understand it to make it work again. :)