But it was jarred loose by recent reading -- and I mused some more --
And I still don't have anything but a beginning. It did occur to me, however, that the wizards of Harry Potter would be a plausible origin for the Fair Folk. People coming out of nowhere with magical powers and taking your kid away.
All right, the age is a little off, and there's no explanation of the changeling myths, but perhaps they took them younger in earlier days -- and perhaps a combination of parking Squibs with Muggle foster parents and taking Muggleborn children got mashed together. (Especially since it's so convenient to explain children you can't stand.)
Still didn't give me anything more than beginning. The only plot idea I can think of is that the Dursley equivalents wise up and go to retrieve their children, but it doesn't appeal to the muse. (Plot bunnies. Free to a good home. . . .)