In SF, we may need to create new species, but in fantasy we have the option not to. This section helps you decide on whether to do it or not.
In Science Fiction
Aside from little green aliens, inventors of SF have no public domain species available. We can’t use Vulcans from Star Trek or Na’-vi from Avatar because someone else owns them. We either have only humans or must invent humanoid (or not) species. Do you want your planet-hopping characters to encounter unique lifeforms on different planets or on other spacecraft? That and aliens arriving on Earth are the only scenario where we must create them, as there are plenty of SF stories with only humans, especially those involving explorers from Earth.
In a universe like Star Trek, some species are ever-present while others are episodic, only appearing in one or two shows. The latter need far less development time. It might be wise to create a few well-developed species (who are part of a crew we use repeatedly) but then spend less effort on everything else. Some ideas might have limited use anyway; rather than discard them, use them for one story and move on. Riskier ideas are well-suited to this because if our audience doesn’t like them, we’re not revisiting them anyway.
A caveat here is that the opposite could happen: we might find ourselves using them more than initially intended, in which case we must be careful not to box ourselves into a corner. Don’t make unnecessary comments in early uses of them, such as, “They never leave their planet.” Unless that’s part of that story, this restriction could come back to haunt us when we want them traveling. There are ways around that, like deciding they’ve been driven from the world, but you get the idea. A side-effect of thinly developed ideas is accidental conflict when we decide to more fully develop them after (italics) publishing them.
Fantasy species are well defined, popular, and mostly public domain. No one can stop us from using them, which is one reason why seemingly everyone does. Does this make them over-used? Are people clamoring for something they haven’t seen? Are you? If so, you could skip to the next section, “Creating a Species,” right now, but there are a few other points to consider.
Is it okay to present the usual species but with minor or significant changes and still call them the same thing? For minor variations, yes (see “What’s In a Name?” in Chapter 1). For more significant alterations, we might want to just strike out farther on purpose and add a new name. Once freed from the original concept via a name, it becomes easier to reimagine an elf or dwarf. Remember the rule of three when using an analogue: at least three changes so people are less likely to realize it’s a modified elf.
Sometimes it seems like we can read ten books by ten different authors and get ten slightly different versions of an elf. Is that good or bad? They’re on different planets, after all, and might develop differently, but it begs the question of why the humans are usually the same as those on Earth. And the horses. And plants.
There is an important caveat to species that aren’t public domain, like Ents and Hobbits from The Lord of the Rings. We can create a very similar species and then give them a different name. The treants and halflings from Dungeons and Dragons come to mind, Tolkien. This has been done for legal reasons, as the original species belong to their creator. We run the risk of legal trouble with this anyway and it goes against the idea of creating something new, but the option remains. It is arguably best to put our own spin on an analogue while renaming it; with enough changes, audiences won’t immediately think of a known species.