Unless this species is the only one in our work, we need to decide how they relate to humans and other species of our invention, or any public domain species we’re using. Inventing this tends to be easier when we’ve created other aspects first, like appearance, habitat, and world view. Ideas can form during those processes. Giving this time and not forcing it can help, as can imagining interactions as if we’re writing scenes between them and others. Then think about why you have them acting the way they do. Use various situations that are likely to occur, such as greetings, farewells, dining etiquette, and how they react to various kinds of news that impacts their fortunes.
We must be sure to decide how two invented species interact. Using elves and dwarves as examples, it’s easy to fall prey to deciding how humans get along with elves and dwarves but not deciding how elves and dwarves get along with each other. A single paragraph is often enough to get started. We’re looking for high level ideas. Anything more detailed can arise while creating stories.
Are they enemies? Friends? Why? Are their legendary battles or animosities? Treaties? Are they allies now but some among them have bad blood? Every species should have opinions and prejudice about others, and humans should feel or think something stereotypical about everything we create (because we do that). There should be classic misunderstandings. Some of our characters should exemplify these ideas while others rise above them. It adds conflict and dimension. We can start with Earth analogues.
Is it normal for one to smash furniture on hearing bad news? Punch a wall? Or react stoically as if nothing has occurred? While these have nothing to do with relationships, the way others perceive them is certainly affected and in turn impacts relationships. If our species literally kills messengers, others aren’t going to send them, for example, unless wanting to get rid of someone that way. More to the point, the species would be see as temperamental by others and this will change how people deal with them. Reputations are born of such things.
A well done species can allow the author to make commentary about humans, which in turn helps our audience relate to our work. We can craft our species to do this. Feel we’re dishonest? Make your species honest. Think we jump to conclusions? Make your species slow and deliberate in its evaluations to the point that it bugs humans. If you think humans are faithful to gods, create a species that is quick to turn its back on gods if not answered, making us look good by comparison. Maybe our species doesn’t understand the concept of property and just takes other people’s stuff like it’s no big deal and we accuse them of being thieves. Any social or cultural expectation of ours can be turned on its head, an opposite expectation given to another species to cause conflict with not only humans, but other species.