Homebrewing creatures

Party composition

The first thing you'll want to keep in mind is the composition of the party. How many players are there? What level are they? What are their strengths and weaknesses? Things like that. By keeping these in mind you'll be able to tailor your custom creatures to fit the party.
If your party is particularly squishy a custom creature with the ability to do multiple hard hitting attacks probably isn't the best if you're looking for a relatively easy or even relatively fair fight.

But you can go a step further than that. If some of the party members are resistant to poison, are able to communicate with animals or have the ability to negate magical damage you can use those elements to either add weaknesses to a creature or make sure the creature can negate these abilities to make the encounter more difficult.
Try to add some diversity whenever possible. If multiple party members have the ability to use a skill of theirs to defeat a creature it could make the fight more dynamic and thus more interesting. A fight that would otherwise be a plain battle of using the same one or two attacks over and over could instead become a fight in which unused skills are suddenly helpful or where multiple skills have to be used in combination with each other for a more powerful attack.

Cosmetic changes

If you need custom creatures because your party knows every creature in the manual and thus how to fight it, you don't have to go much further than simply changing the appearance of that creature. Consider the following three example descriptions:
- In the back of the cave you see a huge, female humanoid. Her serpentine body slithers toward you as she and countless snakes on her head hiss at your sight.
- You hear a thumping sound of hooves behind you. As you turn you stare into the dark, furious eyes of a huge, muscular humanoid with a bovine face and it looks very, very angry.
- A deep, feline growl echoes from deep within the cave, followed by a serpentine hiss. A little while later a blood curling bleat echoes through the caves and the shimmer of horns and three pairs of eyes can be seen faintly in the distance.

If you're familiar with mythological beings you can probably tell these descriptions are about a gorgon, a minotaur and a chimera respectively. A player would likely know this as well and as a result might handle each creature a specific way. For example, shielding your eyes from the gorgon would be for the best unless you'd like to become a statue.
However, I could change the gorgon into a spider-like creature for example and suddenly you no longer expect it to have a petrifying gaze, at least not until it could be too late.

Customizing the way a creature looks goes beyond simply disguising an ability though, it can add a theme to the creature and its surroundings as well. For example, an undead skeleton would look different in a swamp compared to one in a frozen tundra or one in an ocean shore cave. If those undead happen to be mages, sorcerers or other magic wielders they might even know different spells or different variants of the same spell. Consider the following examples as an illustration of this:
- A bolt of brown energy shoots from its mud and lichen covered arm. The energy hits the ground you're standing on and you can feel yourself sink into a quicksand-like substance. You can vaguely make out a smirk on the undead's face, half hidden behind a thick, marshy leaf.
- The undead raises its glistening arm and you see icy vapor rise from its hand. Suddenly bolts of ice erupt from the ground around you, trapping you inside their icicle claws. The undead smirks, causing pieces of ice to break off of its face.
- Bolts of blue energy and a terrible smell of rotten fish erupt from the barnacle covered hands of the undead. The ground beneath you sinks deeper into the water, you can feel the water soak your boots and reach your knees. As the undead smirks you can see small crustaceans hide in the many cavities of the undead's skull.

The previous 3 examples all had the same base creature (some kind of undead mage), the same action and reaction (cast spell and smirk) and the same spell (some form of entanglement), but they all felt different and were themed according to their surroundings.
Doing this not only makes for more interesting encounters, it also helps with making the same types of encounters feel less the same and thus less monotonous.

Switcheroo

Just as you can swap or otherwise alter the appearance of a creature to make it different, so too can you swap the stats and abilities of a creature with another to increase or decrease the difficulty of this creature.
This doesn't work for every creature however, some creatures have a very specific attack or characteristic that would break other creatures or ruin this creature if replaced, but it can work great for most creatures. Need a very strong boar? Give it the stats of a dire bear. Need a weak dragon? Give it the stats of a feathered serpent, a naga or even a large snake.

In some cases you can use creatures with similar powers to scale the other up or down. For example, gorgons, basilisks and cockatrices all have the ability to petrify their victims, but a basilisk is significantly weaker than a gorgon and a cockatrice is weaker still. So swap their stats to make one stronger or weaker, depending on which way you go. There's more which could come into play when making creatures weaker or stronger though, but more on that later.

Biggest threat

Let's delve into altering an existing creature. The first thing you want to figure out is what the biggest threat and the biggest weakness of that creature is. Knowing these two is usually enough to alter the creature enough for whichever encounter you need it, but it also helps with avoiding making a creature too strong or too weak.

Let's take the gorgon as an example again. The biggest threat is their ability to petrify their victims and their biggest weakness is usually a relatively low defense. An encounter with a gorgon is usually one that ends as soon as possible in order to minimize the amount of opportunities the gorgon has to petrify the party members. If you give it more armor or health points the fight will last longer and the chances of ending up with a statue version of a party member increases. So instead you might want to give the gorgon a strong secondary attack while keeping its armor and health points low. Allow it to be able to take a big chunk of health from a party member in one turn, but keep it weak enough to make sure the encounter will likely not take too long and to make sure this health loss won't be a huge issue to deal with.

Favoring stronger attacks over more hit points is usually the way to go, but it does depend on what you need from that specific encounter. In some cases you might need to alter the attacks by adding cooldowns or stages. For example, the gorgon's petrifying gaze might only turn a victim into a statue after it failed their save roll twice instead of once or only when the save is failed by more than a certain amount of points.
Alternatively, increasing a creature's hit points could be exactly what you need. Big, bulky bodyguards might stand in the way of the party and the escaping mob boss. If the party wants to have any chance of catching that boss they'll have to spend every skill they have to burst through these hulking masses of muscle or else it'll just take too long. In other words a timed event against a lot of hit points could be a perfect way of making sure the party spends a lot of their skills before the main boss fight, giving the enemy a slight edge.

Adjust abilities

I want to talk about adjusting abilities a little more. While adding cooldowns or stages works for some abilities, in many cases it does not. So what other options do you have? As far as increasing the difficulty goes you could turn single target spells into multi target spells or even area of effect spells. Enhancement abilities might get a bigger bonus, a creature's speed could be increased as could the amount of attacks they could make in a single turn. Resistances could be enhanced or added, immunities could be added and so on.

If you need to make a creature weaker you could scale down the damage they do, change their weapons, lower their spells, give them a limited amount of spell slots, give them weaker armor, lower their bonus stats, make them roll with disadvantage and so on.

Now you might wonder which of the many options to pick. The answer depends entirely on the circumstances. Who is your party fighting and where? Who and what does your party consist of? What are their strengths and weaknesses? This is where party composition comes into play a lot more.

Milestones

You could add a further layer of weaknesses and strengths via milestones. If the party has done x amount of damage to a creature they might gain a new ability, start using one they hadn't before or lose the benefit of another.
You've done damage equal to 10% of the hydra's total health. Congratulations! It loses a head. Oh, wait. Two more grow in its place. Good luck!
That dire bear is nearing death, it's getting desperate and adrenaline is now really kicking in. You might want to avoid getting hit even more now.

The hydra example is one players will generally accept more easily than the bear example. The latter can feel unfair and against the system's rules, while the prior seems fair considering this is just what the creature is known to do. But a bear would get desperate and adrenaline and fear would course through its veins, so technically that's realistic as well, right? Well, yes, but it's not always that simple. Most game systems have some rules in place and as such the players have certain expectations. If you want to homebrew creatures in a similar fashion to the bear example you're probably best off discussing this with your group first.

Age, disease and weakness

Earlier I mentioned how there's more which could come into play when making a creature weaker or stronger, much of this is related to the appearance of the creature. I don't mean a themed appearance in the same sense as the undead examples from before, but more in the sense of the health of the creature.
A very young, very old or a sick creature could make for a perfect weaker version of a regular creature. A young minotaur might lack the strength to wield a giant axe and a very old one may be too tired to be truly enraged for example.

By playing around with the health of a creature you can make sure the encounter becomes more realistic from a story point of view. If you really want your characters to fight a dragon at level 5, but if this is normally saved for level 15 you could throw in a very old or diseased dragon. Simply having a dragon with lower stats, but otherwise seemingly healthy doesn't usually make sense and can make later fights with other dragons seem less special. At the same time killing a diseased dragon could lead to all sorts of story threads as well.

Starting from scratch

If you're creating a creature entirely from scratch and are unsure what stats and abilities to give it you could look for other, similar creatures in a monster manual or similar source of monsters. But this could be a lengthy process or an impossible one if you have a particularly unique creature. So what then?
The simplest way of giving this creature stats is to check what you need from this creature and to see what it could logically have based on its physiology.

Does your creature need to survive a fight for a while and does it have a thick skin? Give it a high armor class or equivalent attribute. Does it not have a thick skin? Perhaps it could have a lot of hit points instead or be very agile and dexterous and thus able to dodge attacks.
What about its attacks? Do they need to deal serious damage or slowly take down a target? Give it stats, weapons and attacks accordingly.

The type of attacks, weapons, spells and other abilities you can give a creature can be tricky to figure out. In most cases it's merely a case of figuring out what your party can handle and perhaps even using averages (average hit, average damage, etc.) to assign attributes accordingly, but if you have a particularly unique attack or ability you might just have to test it in a few rounds of combat. Obviously you want to test this outside of the campaign. Simply rolling the dice yourself will do the trick.

Avoid a TPK

Whenever you make a creature stronger you add to the risk of a total party kill (TPK) or even the death of just one player character (PC). If you want to avoid this you might want to test your creation a little before unleashing it on your players. What is the max damage they can do? What is the average damage they can do? How likely is it the party will survive if they happen to have a string of bad luck leading to lower rolls?
You can't always prepare for everything, but you can prepare for the worst in this case. Nothing's worse than killing a PC or an entire party when you didn't intend to, especially if you do so with your own created creature. But don't worry, unless you changed things significantly in terms of difficulty you probably won't encounter a TPK.