Now, in fantasy you can with suitable application of magic get yourself out of the situation where sky-high infant mortality and short life-expectancy. This, however, merely gets you into the situation you often find in SF. You still need the 2.1 children per couple to keep up with replacement rate.
Some writers weasel out of it with immortality. And I do mean weasel out of it. Though there are other reasons for immortality in fiction -- some aesthetically sound, others not -- I have read works in which it was blatantly obvious that the reason people were immortal was to justify draconian population control measures. SF writers boasted of having foreseen the population crisis decades before it was taken seriously outside SF; SF writers not only did not foresee demographic collapse, they are not writing about it even now that it is not only taken seriously outside SF but actually occurring.
Indeed, some of them have population control measures in societies that would be suffering demographic collapse even without such measures. Beta Colony, for instance, tracks most closely to a Western Europe welfare state minus the lumpenproletariat, which means it would, logically, have a sub-replacement birthrate. Yet it forces people to attend classes to be licensed to have a child, and then you have to pay to have a second, and still more to have a third -- all of which, naturally, would depress the rate still farther. This would only work if people have an obsessive desire to have babies. Yet no one evinces such obsessive desire. Cordelia's reflections on the possibility of having a child are not very certain about it, and she doesn't reflect that she's odd, that most other people structure their lives about the possibility.
And most people would structure their lives about it if desperately wanting to have babies was the condition of reproduction, and within very few generations. Philoprogentiveness would be selected for, whether it was a genetic or cultural factor. And selected for rather in the manner that hunger is selected for. When children can be accidents, sex drive will cope with a lack of desire for them, but when they have to be planned, it can't do it alone. (Which will not be all good for children. Battered children are more likely to be planned than non-battered children. They have the kid to fulfill their desires, and if the kid doesn't fulfill them -- whap.)
This is why contraception does not completely change women's relationship to childbearing. The children still have to be born, or society will collapse within -- well, within a lifetime.
Without this folly, age, and cold decay:
If all were minded so, the times should cease
And threescore year would make the world away.
To completely change it, you would have to have artificial conception and incubation and mandatory sterilization of everyone -- probably with mandatory abortions for the accidents. And then there is the little question of how the children get raised. . . .
Update: I am now screening comments to this to prevent its going off on tangents. Even non-tangential things will get unscreened as I have time to deal with them.