Homebrewing classes

Why?

Before you start creating your own class you'll want to ask yourself why you wish to create one. Does your gaming system lack something? Are there game mechanics lacking that prevent you from creating something in particular? The answer matters, as many systems already provide enough materials to create all sorts of classes, all that is required is giving them a different flavor. Why does this matter? Because creating a new class that does the same as an existing class could mean stepping on the toes of those who wish to play the existing class. More on this later though.

Another reason to consider the why is because some concepts work better as a new race or as an extension of an existing class. A knight, samurai, gladiator and a Roman centurion can generally all be created using the same class, but with different archetypes. They can often already be created within the same system by simply wording things differently, picking specific feats, multiclassing or using any of the other mechanics readily available.

So, generally speaking, creating a class requires creating something that adds something new to a gaming system. This depends on your personal desires and what your group is comfortable with of course, but, from a balancing point of view, this is something you'll want to keep in mind as homebrewing a class is difficult enough as it is.

Difficulty

Speaking of the difficulty, how difficult is creating a class really? Surely you could just take one class, rename everything, change the abilities a little, and you're done, right? Not really. In most games the classes have been tested and created with care, messing with it can lead to a wide range of balancing issues. Many classes are also built around certain mechanics, messing with those in any way will cause a ripple effect through the entire class.
Creating a new class means creating a concept from scratch or altering a new one to the point of it essentially being something new entirely. Anything else would work better as a minor archetype, race or other, smaller mechanic. But all this doesn't mean it's impossible to create a new class of course, not at all.

Points and averages

Let's delve into the creation process. A good place to start from are the averages of all the classes in a specific gaming system, and by this I mean the averages of everything. The average health pool, the average amount of abilities, the average damage output, and so on. This means you'll probably have to figure these parts out, but for some of the bigger systems they can be found online or in official books.
Once you've figured out the averages you can build from them. Is the average hit-die a d8? Maybe up it to a d10 if your class is supposed to be more of a tank. Is the average amount of spells for those classes with spells 5 at first level, but your class is more of a hybrid? Lower them to 3.

Of course, you'll want to make sure there's a balance. An easy way to keep things balanced is to assign points to everything. Similar to how the skill point generation part of character creation works in some systems. A d6 hit-die could be 1 point, a d8 2, and a d10 4 points, for example. Then give everything else points too, check how many points each existing class has, and go from there. This is a little tricky to get right, as it can be difficult to quantify some abilities in terms of points, but it's definitely a good start. Also make sure you set a minimum and maximum amount of points you have to give any specific part of a class or you'll end up with balance issues. Having a d20 hit-die at the cost of a proper spell pool isn't really balanced, for example.

Purpose

Now before you start min-maxing your class abilities, try to think of what the purpose of your class is supposed to be. A damage soaking tank will generally have a bigger hit-point pool, but a lower damage output, while a powerful wizard will often be a fragile glass cannon. But it goes further than that. Supporting roles come into play too, ranging from healing and buffs to stealth and scouting, and from social encounters to travel. These elements depend on the system you're working with of course, some systems are focused far more on combat and less so on other elements for example, but it's worth keeping in mind nonetheless. Each class usually has some minor elements for flavor as well, like a bard's charms or a warlock's demonic elements. These elements aren't always hugely important and can often be done by other classes too, but not always to the same effect.

Once you've found the purpose of your class it becomes a lot easier to assign the points and use the averages to create something unique. But it doesn't end there of course, there's far more to be done. Including math for new abilities, perhaps writing spell descriptions if you feel like it, figuring out how abilities work (materials, spoken vs. unspoken, etc.), and anything else that is part of your game system and applicable to the way you play.

Number crunching

Time for a part you'll likely either hate or love: number crunching and math. While technically you could just copy abilities and reword them, which is also the safest way of doing things, this isn't always the most fun way. Creating your own abilities with your own damage values can be incredibly satisfying, but it can also lead to a lot of balance issues.
Using the averages again, we can build upon the existing to make sure the first draft is at least in the ball park of balance, but new abilities will usually require fine tuning over time.

The biggest thing to keep in mind is the minimum and average damage an ability can do, especially the average. Making sure the maximum values are roughly equal is very easy after all. For example, replacing an ability that does 1d12 damage with one that does 2d6 damage creates an imbalance. 2d6 means at least a minimum of 2 damage, but, more importantly, means the average roll is higher, but it also means the probability of rolling something other than the lowest or highest values is different.

If you were to roll a d12 1000 times, it would land on 1 about 83 times (on average), on 2 83 times, on 3 83 times, and so on. If you rolled 2d6 1000 times, the result of the two dice would be 2 only about 27 times, 3 about 55 times, 7 about 166 times, and 12 about 27 times as well. So in other words, with 2d6 you'll throw a 7 twice as often as with a d12, but a 2 or 12 three times as fewer as with a d12. The total value of these 1000 rolls will also be roughly 500 points higher for the 2d6 rolls compared to the d12.

Of course, the differences are often minimal, but for larger abilities they can make a big difference, not to mention the effect the smaller differences will have over time. The difference between the average d12 roll and average 2d6 roll is only 0.5, but when different effects and other elements come into play, these small differences can snowball into something bigger.

But there's a psychological element to it too. The 2d6 rolls are more steady, while the d12 is more volatile. This can change the way an ability feels to a player. With a 2d6 roll you know the chances of a decent roll are, well, decent, whereas with a d12 roll you know it could be all or nothing. This can have effects in terms of play style, how a specific ability feels, the sense of realism in terms of what you'd expect from a specific ability, and so on.

Quick note, to get the average result of any dice roll, use the following: Maximum value of a die + 1 / 2, then multiply the result times the amount of dice. So for a d12 it would be 12 + 1 / 2 = 6.5. For 2d6 it would be (6 + 1 / 2) * 2 = 7.
There are many tools for this available online, as well as tools for the probabilities of every dice result you may need to help you with this process.

Brand new abilities

But what if you want to create a brand new ability, one you can't really base on existing abilities? Well, the chances of this are very low, but it raises the question whether the game system you're using is right for whatever you're trying to create. But let's say it is right, chances are there is still something available to use as a basis for whatever ability you wish to create. Ignore all the flavor and descriptions of the existing abilities, and just look at what they do mechanically. Buff a character, take or add health, create or solve a roll against another character, and so on. Most systems usually cover their bases, so it's unlikely you'll find something that hasn't been covered mechanically. While you might not find rules for laser pistols in a D&D book, it is essentially still an attack with a projectile weapon, possibly with or without charges as you see fit.

If you can't find a basis within the system you're using, you might want to rethink what you're trying to create. A class will always have a basis of some sorts, even if it's something as simple as a minimal character sheet, and abilities are always covered in some shape or form too. So if you're trying to cover something that isn't covered mechanically by these elements, you might be trying to create something that isn't a class, but perhaps a regular game mechanic that could apply to all classes.

Test and test and test

This should go without saying, but test your creations. Abilities especially require a lot of testing under different circumstances. Creating your own battles or circumstances is a great way to work out the first few flaws, but some flaws will only reveal themselves in actual play. Don't worry too much about whether things are perfect though, as long as you're having fun with your group that's all that really matters.

The testing and tweaking phase is usually also the longest phase. For some it can take a full week, others spend a little time here and there over the course of several months or more, there's no real way to predict how long this stage will take, there are simply too many variables.

This is also the stage where min-maxing is definitely a good thing. You'll want to see the effects of what the best possible stats do for this class, as well as what the worst possible stats will do. Does it break things in any direction? Is the version with the worst possible stats somehow still quite powerful?
Test your class with different races too, assuming they're part of your game system. Racial traits can sometimes break specific elements or make them redundant. If your class grants a feature already covered by an existing race, maybe think about removing it or changing it, as it'd mean this combination isn't all that balanced.

Another test you could do is put your class up against whichever class (or classes) is the most similar to yours. If you created a magic-focused class, put it up against a different magic-focused class, but make sure the overall class is still similar. There's no point putting a glass cannon up against a healer, for example, as they serve different purposes.
The point of this test is to see if one comes out on top a lot more than the other. If this is the case, there's probably some tweaking needed on your class.

Guidance

Homebrewing classes isn't something new, it's been done over and over again for the bigger game systems, and some even have their own tips and tricks. Turning to those official source books can be of great help, but it doesn't have to stop there. Guides like the one you're reading right now could be of help, experiences of people found all over the internet could be helpful, and even books of other game systems could be of help. Creating a class can be incredibly complex, which means losing track of some of the smaller details can be very easy. Taking everything one element at a time is always a good way of doing things, but allowing other people or sources of help to guide you or to keep track of the minor elements can be incredibly helpful too.

Help can also come from sometimes less foreseen directions. Rules on multi-classing, for example, offer great ways to mix different elements of existing classes in a balanced way. This could offer a good foundation for your own custom class.
One potentially overlooked direction is existing creations. If you're working with a popular game, chances are somebody already created what you have in mind, for better or worse. At the very least it can offer you some inspiration or tips and tricks.

Altering the existing

Lastly, sometimes it really is just both easier and better to alter what already exists. Rewriting the descriptions, changing a few of the abilities or mixing them between classes, adding some background or flavor text, and you're done. Even adding elements to all existing classes to cover a wider base could be an option.
Depending on your group, creating a new class could mean making others redundant or otherwise less special. Adding a ninja class to D&D could make the rogue class less special, for example, as they'll likely be very similar. If this means somebody in your group, including the GM (Game Master) who might have plans for future rogue NPCs, has less fun or feels like you're stepping on their toes in any way, it's probably for the best to just use a rogue and play them differently. In this particular case it's definitely possible to play a ninja using the rogue class, but it goes for all classes.

I've said it before, but at the end of the day it's all about fun. So do your thing, have fun with it, and try to make sure your group members have more fun because of it too.