Author's Note: This is an adaptation of a popular but highly controversial journal entry I wrote here almost two years ago. This is solely my own opinion and is in no way intended to be representative of my employer. Enjoy it (or hate it!) for what I meant it to be - a sarcastic backlash against a common aspiration of game development newcomers.
It's nothing new, this trend, but it also doesn't seem to be going away as quickly as most fads; perhaps we can accuse World of Warcraft for making this such a huge deal, or maybe it all goes back to Ultima Online and Everquest. We may never know for sure.
One thing is definitely clear, though, and that's that a significant contingent of people show up here at GDNet wanting to make their own MMO games. There have been many words bandied about in the past over this subject, and plenty of advice warning newbies not to get their hopes up when it comes to shipping a true MMO.
Sure, not everyone is committed to the RPG genre; some people want to make an FPS, or maybe even an RTS. Others might have even more creative and unique ideas for how their game should play. But the real issue here is not the gameplay itself: it's the MMO part. More than that, it isn't even the "multiplayer" or "online" that is really an issue - it's the "massive."
MMOs are expected to host dozens of servers running thousands of players apiece. Successful games may be played by over a million people; some notable ones are played by far more than that. Even a low-grade MMO serves a few hundred thousand players.
With modern technology and toolkits, it's entirely in reach of a lone wolf or a small team to create an incredible game that can host even a few hundred players. However, as wiser men than myself have observed, "a difference in quantity can often mean a difference in kind."
Consider the original airplane, as developed by the Wright brothers over a century ago: a simple wood and canvas structure with a (relatively) lightweight and effective engine, with some simple control mechanisms. Now consider a jet liner. Both are flying machines; but in order to comfortably handle two hundred passengers, the airplane had to do a lot of changing over the years.
Even a fighter jet is a different beast. Just changing one quantity - speed - leads to qualitative differences in many ways. A fighter jet has different engines, different control surfaces, different kinds of fuel, different training to fly, and on and on. Go one step further: a fighter jet that's invisible to radar again has many, many differences (ranging from shape to materials to operating principles) than a traditional jet.
Many people have set eyes on the available game-development technologies of today, and noticed that they can put a hundred players in a world and turn them loose. The assumption that you can just twiddle some knobs and tweak some sliders to get an MMO is deeply flawed.
As a personal anecdote, I once set out to write a realtime ray-tracing engine. I had experience with a popular graphics package called POV-Ray, back when it was still a relatively fresh spinoff from DKBTrace. Since POV-Ray represented an early incarnation of a model of what we would now call "open source", I had access to the code; my plan was to "optimize" POV-Ray and turn it into a realtime system.
What I failed to understand was that the difference in speed I wanted represented a tremendous change in how the code had to be written. I managed to eke out about a 5% speed gain over the original POV-Ray before giving up and writing a fresh engine from scratch.
(For the record, I did achieve realtime ray-traced rendering in that project, but it was truly hideous and awful code. I still have it someplace but if I remember correctly I left it in a horribly broken state. It's probably better that way. Overall, though, the experience was incredibly educational.)
It's a far cry from 16 people playing Halo on a home-rigged LAN to the kind of stuff a real MMO must cope with. Even throwing in something like Xbox Live Matchmaking is incredibly resource-intensive. Companies like Bungie have people whose full time jobs involve doing very little but making sure the matchmaking experience runs smoothly. When a major company has to employ people to do nothing but run their multiplayer system, you can bet it's a good sign that Little Old Joe in his garage or basement isn't going to be up to doing something similar.
But wait! Even running a matchmaking system for a few hundred thousand players pales in comparison to what an MMO must do. After all, a matchmaker simply needs to join up different players and get them running on a separate system (usually hosted by one of the players themselves, or maybe a dedicated server for PC games). If dedicated servers are involved, the matchmaker is often really just a catalog of what severs are running. This is the kind of thing that a lone wolf coder could easily accomplish on his own; something like Xbox Live matchmaking, however, is getting past the realm of stuff that one guy (or gal) could ever do alone.
So if a solo developer can't really hope to build, deploy, and run a large-scale matchmaking service, what about an MMO?
Let's take the demands of a typical matchmaking service and chuck in a handful of complications.
Here's an educational homework exercise for you. Even better, it has a magic trick twist to it. Pick your favorite MMO, and go look up the game credits, either in-game or on the web or wherever. Count the number of people involved. Don't cherry-pick either - look at programmers, artists, musicians, producers, designers, marketing folks, operations/IT staff, and anyone else you can find who participates in a non-trivial way in making that game happen. (You can ignore janitorial staff for the purposes of this exercise.)
Now, don't tell me your number. Keep it secret.
Without talking to you in any way - before you even do this exercise - I will know more about your number than you might be comfortable admitting.
In fact, I'm so sure about this number, that I'll even make you a bet.
I will bet you a beer (or suitable beverage of your choice) that you can't find an MMO with over a million players with a development team of less than 100 people.
Here's another magic trick. Go find the lead developers and senior-level programmers, artists, producers, musicians, and other high-ranking folks from your MMO's team. Now go find out what other games those people worked on before the MMO. Do the same thing for the less-senior guys as well; average up how many titles everyone has worked on in their past.
I will bet you a second tasty beverage that the average developer working on an MMO has shipped at least 2 games prior to shipping a successful MMO. Many successful MMO titles are even the results of collaborative efforts from dozens of people with prior MMO experience.
At its peak, the Guild Wars 2 team included something like 300 people - and that doesn't account for those who came and left over the course of the game's development. The codebase is many millions of lines of code (I don't have exact figures, and probably couldn't release them anyways.) There are terabytes of assets - ranging from artwork to music to design documents to configuration files.
If that scale alone isn't enough to convince you that MMOs are out of reach for the Five Amigos, then let's talk business.
MMOs are fiercely competitive, and a cutthroat business. People are doing all manner of wildly innovative and devious things to make money in the MMO space, and most of them are failing. There are some fascinating studies of how people move between MMOs, and they tend to suggest that many people are constantly on the hunt for the next great big thing - meaning that they'll bail on a mediocre MMO in a heartbeat to go play something cooler.
Running an MMO is immensely expensive. Internet hosting and server costs alone can be in the tens of thousands of dollars a month range. Buying all the hardware you need to run the game up-front can be well into the millions. You need a dedicated datacenter for the endeavor, with redundant power, fire safety systems, industrial cooling, and hundreds of miles of both copper cabling and fiber optics. A single network switch capable of running an MMO backbone can cost ten grand by itself. And if you want a global reach, you'd better roll out a datacenter on every major continent, at the very least.
I'll gloss over the details like how to go about creating all the assets and content necessary to flesh out a true MMO, and stick with the technical specifics (which is where I spend most of my time anyways).
At this level, everything becomes important.
The difference between allocating space for a buffer on the stack versus the heap is a crucial design decision. Calling conventions must be selected with great care. Exception handling is nearly verboten, because it has too much runtime overhead and can create unpredictable execution flows in code that must run continuously for years at a time. One-byte memory leaks are show-stopper, can't-ship-the-game-like-this bugs - because that one byte will kill you when it leaks constantly for eight months. Microseconds of sloppily implemented code turn into milliseconds of delay when scaled across thousands of players, which adds up to lag - the number one evil of an MMO.
You'd better be a sheer wizard at lock-based concurrency, because the order in which you lock things might spell the difference between a happy server and a worldwide outage. For that matter, you'd better be a wizard at lockless concurrency too, because locks are often too expensive and too exclusive to scale to thousands of active connections. Memorize those atomic operations, folks, and if you still think "volatile" means anything useful, go back to Go and do not collect $200.
You'd better understand how operating system kernels work, because you'll be taxing the living hell out of yours. Your job will bank on you knowing things like pre-emptive multitasking architectures, monolithic versus micro-kernels, how kernel-versus-user-mode code works, and how schedulers are implemented.
Better brush up on your compiler theory, too, because you'll need to know exactly how your code gets converted to machine language. For that matter, you'd better know your assembly language and your computer architecture, too. Two or three clock cycles here and there will add up in a damned impressive hurry. Don't know what a superscalar CPU architecture is? Pipelining? L1, L2, and L3 caching? Don't know how a memory bus is implemented? Don't know the difference between a northbridge and a southbridge? Don't know the role of device drivers and how they work? Don't know where the network stack of your OS of choice lives and under what context it runs? Pack your bags and don't sell the bike shop, 'cuz you aren't cut out to be writing an MMO server.
It takes half a dozen dedicated programmers just to write the servers for Guild Wars 2 - and that doesn't include the people who wrote some of the legacy code we inherited and modified from GW1, or the people working on auxiliary support features like web sites, forums, wikis, and so on. It doesn't even include the network admins who build the hardware and maintain it, or the community managers who take care of ensuring that the experience is enjoyable for all the players.
The guys I work with are, to a soul, incredibly good programmers. I'm still not entirely convinced that I'm cut out to be working with them, or on a project of this magnitude. It's without question an extreme challenge - but one that I'm tackling with gusto, because I love a good challenge. There's more brilliance packed into the room I sit in every day than I've seen in years in the industry, and I've worked with some damn good people in the past.
My goal here isn't to scare off everyone who doesn't have a pile of money and time at their disposal; my goal is to filter down to the people who are motivated enough to put in that time. I'm not trying to discourage people from pursuing their dreams - I'm just trying to help people self-select their dreams and maybe try leaping at some smaller goals before shooting for the moon.
Individuals interested in accomplishing great stuff need to first prove themselves - and hone their skills - in smaller arenas. Some may not agree, which is fine, but harsh experience and observation of many successful people in the business (as well as many people who failed to go anywhere) has taught me that the only certain way to large-scale success is to master the small first. Yes, maybe some obscure genius did the big thing once without ever learning the basics, but those are exceptions, and rare ones at that - in my mind, it's a disservice to newcomers to any craft to tell them that they can be just like that, because in the majority of cases it simply isn't true.
I'm not arguing that individuals or smaller teams can't make awesome games - there's ample evidence that they can and do. Even multiplayer games are not out of reach. My point centers entirely around the "massive" label.
By all means - go make games. Make fun, awesome, compelling, memorable, and successful games.
Just be mindful of the fact that success is equal parts intensely hard work, and knowing how to pick your battles.
GameDev.net Soapbox logo design by Mark "Prinz Eugn" Simpson
The MMO Gold Rush
It's nothing new, this trend, but it also doesn't seem to be going away as quickly as most fads; perhaps we can accuse World of Warcraft for making this such a huge deal, or maybe it all goes back to Ultima Online and Everquest. We may never know for sure.
One thing is definitely clear, though, and that's that a significant contingent of people show up here at GDNet wanting to make their own MMO games. There have been many words bandied about in the past over this subject, and plenty of advice warning newbies not to get their hopes up when it comes to shipping a true MMO.
Sure, not everyone is committed to the RPG genre; some people want to make an FPS, or maybe even an RTS. Others might have even more creative and unique ideas for how their game should play. But the real issue here is not the gameplay itself: it's the MMO part. More than that, it isn't even the "multiplayer" or "online" that is really an issue - it's the "massive."
The Real Issue
MMOs are expected to host dozens of servers running thousands of players apiece. Successful games may be played by over a million people; some notable ones are played by far more than that. Even a low-grade MMO serves a few hundred thousand players.
With modern technology and toolkits, it's entirely in reach of a lone wolf or a small team to create an incredible game that can host even a few hundred players. However, as wiser men than myself have observed, "a difference in quantity can often mean a difference in kind."
Consider the original airplane, as developed by the Wright brothers over a century ago: a simple wood and canvas structure with a (relatively) lightweight and effective engine, with some simple control mechanisms. Now consider a jet liner. Both are flying machines; but in order to comfortably handle two hundred passengers, the airplane had to do a lot of changing over the years.
Even a fighter jet is a different beast. Just changing one quantity - speed - leads to qualitative differences in many ways. A fighter jet has different engines, different control surfaces, different kinds of fuel, different training to fly, and on and on. Go one step further: a fighter jet that's invisible to radar again has many, many differences (ranging from shape to materials to operating principles) than a traditional jet.
Many people have set eyes on the available game-development technologies of today, and noticed that they can put a hundred players in a world and turn them loose. The assumption that you can just twiddle some knobs and tweak some sliders to get an MMO is deeply flawed.
As a personal anecdote, I once set out to write a realtime ray-tracing engine. I had experience with a popular graphics package called POV-Ray, back when it was still a relatively fresh spinoff from DKBTrace. Since POV-Ray represented an early incarnation of a model of what we would now call "open source", I had access to the code; my plan was to "optimize" POV-Ray and turn it into a realtime system.
What I failed to understand was that the difference in speed I wanted represented a tremendous change in how the code had to be written. I managed to eke out about a 5% speed gain over the original POV-Ray before giving up and writing a fresh engine from scratch.
(For the record, I did achieve realtime ray-traced rendering in that project, but it was truly hideous and awful code. I still have it someplace but if I remember correctly I left it in a horribly broken state. It's probably better that way. Overall, though, the experience was incredibly educational.)
The Challenges
It's a far cry from 16 people playing Halo on a home-rigged LAN to the kind of stuff a real MMO must cope with. Even throwing in something like Xbox Live Matchmaking is incredibly resource-intensive. Companies like Bungie have people whose full time jobs involve doing very little but making sure the matchmaking experience runs smoothly. When a major company has to employ people to do nothing but run their multiplayer system, you can bet it's a good sign that Little Old Joe in his garage or basement isn't going to be up to doing something similar.
But wait! Even running a matchmaking system for a few hundred thousand players pales in comparison to what an MMO must do. After all, a matchmaker simply needs to join up different players and get them running on a separate system (usually hosted by one of the players themselves, or maybe a dedicated server for PC games). If dedicated servers are involved, the matchmaker is often really just a catalog of what severs are running. This is the kind of thing that a lone wolf coder could easily accomplish on his own; something like Xbox Live matchmaking, however, is getting past the realm of stuff that one guy (or gal) could ever do alone.
So if a solo developer can't really hope to build, deploy, and run a large-scale matchmaking service, what about an MMO?
Let's take the demands of a typical matchmaking service and chuck in a handful of complications.
- Persistent Worlds. Instead of just shuffling through a list of players and handing out matchups, which may last a few rounds at the most, an MMO must store data on every player and every action ever made. Sure, many modern games store statistics like how many shots you fired, where you died most often, and so on; but even that is a huge leap of data warehousing beyond what a typical small team can hope to build and maintain by themselves.
- Active Simulation. A matchmaker can often get away with having a player host the game, which is the typical approach used in console games these days. Matchmaking servers don't have to have any code besides the stuff that links people together, making the actual multiplayer portion fairly straightforward by comparison. Again, remember that for up to about a dozen players, off-the-shelf network code is often sufficient to get a decent game experience. It does require some skill to make it happen, but it's at least plausible that a single person (or a small team of people) could get away with building a good low-scale multiplayer game.
- Massive Scale. We're not talking about sorting through a few thousand people to create a 10-man match; we're talking about having thousands of people connected to a single server, all constantly doing things. Combine this with persistence and active simulation, and stuff just got a lot more painful.
An Amusing Magic Trick
Here's an educational homework exercise for you. Even better, it has a magic trick twist to it. Pick your favorite MMO, and go look up the game credits, either in-game or on the web or wherever. Count the number of people involved. Don't cherry-pick either - look at programmers, artists, musicians, producers, designers, marketing folks, operations/IT staff, and anyone else you can find who participates in a non-trivial way in making that game happen. (You can ignore janitorial staff for the purposes of this exercise.)
Now, don't tell me your number. Keep it secret.
Without talking to you in any way - before you even do this exercise - I will know more about your number than you might be comfortable admitting.
In fact, I'm so sure about this number, that I'll even make you a bet.
I will bet you a beer (or suitable beverage of your choice) that you can't find an MMO with over a million players with a development team of less than 100 people.
Here's another magic trick. Go find the lead developers and senior-level programmers, artists, producers, musicians, and other high-ranking folks from your MMO's team. Now go find out what other games those people worked on before the MMO. Do the same thing for the less-senior guys as well; average up how many titles everyone has worked on in their past.
I will bet you a second tasty beverage that the average developer working on an MMO has shipped at least 2 games prior to shipping a successful MMO. Many successful MMO titles are even the results of collaborative efforts from dozens of people with prior MMO experience.
A Case Study in Real MMOs
At its peak, the Guild Wars 2 team included something like 300 people - and that doesn't account for those who came and left over the course of the game's development. The codebase is many millions of lines of code (I don't have exact figures, and probably couldn't release them anyways.) There are terabytes of assets - ranging from artwork to music to design documents to configuration files.
If that scale alone isn't enough to convince you that MMOs are out of reach for the Five Amigos, then let's talk business.
MMOs are fiercely competitive, and a cutthroat business. People are doing all manner of wildly innovative and devious things to make money in the MMO space, and most of them are failing. There are some fascinating studies of how people move between MMOs, and they tend to suggest that many people are constantly on the hunt for the next great big thing - meaning that they'll bail on a mediocre MMO in a heartbeat to go play something cooler.
Running an MMO is immensely expensive. Internet hosting and server costs alone can be in the tens of thousands of dollars a month range. Buying all the hardware you need to run the game up-front can be well into the millions. You need a dedicated datacenter for the endeavor, with redundant power, fire safety systems, industrial cooling, and hundreds of miles of both copper cabling and fiber optics. A single network switch capable of running an MMO backbone can cost ten grand by itself. And if you want a global reach, you'd better roll out a datacenter on every major continent, at the very least.
I'll gloss over the details like how to go about creating all the assets and content necessary to flesh out a true MMO, and stick with the technical specifics (which is where I spend most of my time anyways).
At this level, everything becomes important.
The difference between allocating space for a buffer on the stack versus the heap is a crucial design decision. Calling conventions must be selected with great care. Exception handling is nearly verboten, because it has too much runtime overhead and can create unpredictable execution flows in code that must run continuously for years at a time. One-byte memory leaks are show-stopper, can't-ship-the-game-like-this bugs - because that one byte will kill you when it leaks constantly for eight months. Microseconds of sloppily implemented code turn into milliseconds of delay when scaled across thousands of players, which adds up to lag - the number one evil of an MMO.
You'd better be a sheer wizard at lock-based concurrency, because the order in which you lock things might spell the difference between a happy server and a worldwide outage. For that matter, you'd better be a wizard at lockless concurrency too, because locks are often too expensive and too exclusive to scale to thousands of active connections. Memorize those atomic operations, folks, and if you still think "volatile" means anything useful, go back to Go and do not collect $200.
You'd better understand how operating system kernels work, because you'll be taxing the living hell out of yours. Your job will bank on you knowing things like pre-emptive multitasking architectures, monolithic versus micro-kernels, how kernel-versus-user-mode code works, and how schedulers are implemented.
Better brush up on your compiler theory, too, because you'll need to know exactly how your code gets converted to machine language. For that matter, you'd better know your assembly language and your computer architecture, too. Two or three clock cycles here and there will add up in a damned impressive hurry. Don't know what a superscalar CPU architecture is? Pipelining? L1, L2, and L3 caching? Don't know how a memory bus is implemented? Don't know the difference between a northbridge and a southbridge? Don't know the role of device drivers and how they work? Don't know where the network stack of your OS of choice lives and under what context it runs? Pack your bags and don't sell the bike shop, 'cuz you aren't cut out to be writing an MMO server.
It takes half a dozen dedicated programmers just to write the servers for Guild Wars 2 - and that doesn't include the people who wrote some of the legacy code we inherited and modified from GW1, or the people working on auxiliary support features like web sites, forums, wikis, and so on. It doesn't even include the network admins who build the hardware and maintain it, or the community managers who take care of ensuring that the experience is enjoyable for all the players.
The guys I work with are, to a soul, incredibly good programmers. I'm still not entirely convinced that I'm cut out to be working with them, or on a project of this magnitude. It's without question an extreme challenge - but one that I'm tackling with gusto, because I love a good challenge. There's more brilliance packed into the room I sit in every day than I've seen in years in the industry, and I've worked with some damn good people in the past.
The Moral of the Story
My goal here isn't to scare off everyone who doesn't have a pile of money and time at their disposal; my goal is to filter down to the people who are motivated enough to put in that time. I'm not trying to discourage people from pursuing their dreams - I'm just trying to help people self-select their dreams and maybe try leaping at some smaller goals before shooting for the moon.
Individuals interested in accomplishing great stuff need to first prove themselves - and hone their skills - in smaller arenas. Some may not agree, which is fine, but harsh experience and observation of many successful people in the business (as well as many people who failed to go anywhere) has taught me that the only certain way to large-scale success is to master the small first. Yes, maybe some obscure genius did the big thing once without ever learning the basics, but those are exceptions, and rare ones at that - in my mind, it's a disservice to newcomers to any craft to tell them that they can be just like that, because in the majority of cases it simply isn't true.
I'm not arguing that individuals or smaller teams can't make awesome games - there's ample evidence that they can and do. Even multiplayer games are not out of reach. My point centers entirely around the "massive" label.
By all means - go make games. Make fun, awesome, compelling, memorable, and successful games.
Just be mindful of the fact that success is equal parts intensely hard work, and knowing how to pick your battles.
GameDev.net Soapbox logo design by Mark "Prinz Eugn" Simpson