You've got to pick your anachronisms. Gotta be some, of course, the essence of steampunk is anachronistic tech. But Victorian era covered a lot of stuff -- do you want to have electric lighting? Less messy than gaslight, some rich houses gave up candles only when they could have electric -- but another name for the genre is gaslight romance.
Plus, of course, if you're going into the future -- biology can get interesting. There's Island of Dr. Moreau if you want to run off Victorian extrapolation. Or you could use some of their more excited proclamations about glands from the era -- in fact, by the time they had electric lights. . . .
If, of course, you are as silly as me and want to do space opera steampunk, things get even more fun.