dalerankdalerank
RU/EN/ITSupport
Abnormal programming

We'll all end up there…

Jul 18, 202611 min

...or why game engines always end up the same way.

Have you ever thought about the fact that the most expensive engine in the industry is written in such a way that it's easier to throw it out and write it again than to understand and fix it? Or that the most successful studio lived for ten years on code that its own authors called "hacked together" — and in Russian there's a more fitting term, starts with "sh..." ends with "...code". Meanwhile an engine that's architecturally correct dies in obscurity because the company ran out of money to develop it. If you didn't recognize the examples, there's an explanation at the end of the article.

Any engine that has "live" games running on it goes through four stages: "works, sort of", "works, but I'm ashamed to show it", "not ashamed to show it, but nobody knows how it works", and "runs on a separate language that we invented ourselves". Then the circle closes and a new turn of history begins, because the team realizes that things can't go on like this.

Let me say up front that this is largely a retelling of other people's thoughts — GDC talks, lectures about patterns, and conversations with colleagues from big studios. The references to words and blogs from twenty years ago aren't here for nostalgia but for a reason: information on the internet doesn't live long, the blogs of the 2000s are already half deleted, yet conclusions from fifteen years ago are still indecently relevant.

Architecture and design are always done by different people

The first thing worth understanding about game engines is that an engine's architecture and its design (in the sense of "systems design") are not synonyms. An architect and a designer are genuinely different roles held by different people, and when that's the case things turn out well — but sometimes one bearded guy combines them both, and then you get what you get, like in that old joke... that if you walk a girl and kiss the car at the same time, both jobs get done badly.

Because architecture is about processes and results, whereas systems design is almost always about decisions and connections. And the main thing in a designer's work isn't writing something good instead of something bad — design begins where both extreme options are good but pull in different directions. That's when the choice has far-reaching consequences that will come back to bite you three years down the road. In the first prototype of an engine everything is usually bad, and the designer here is basically a fifth wheel rolling alongside, actively getting in the team's way. The designer's job is to make sure that along every axis of development there's minimal resistance when writing code and creating content. Code and content sit at opposite poles of development, so very often this is hard, and sometimes impossible.

Granularity:   flexibility <───────●──────> simplicity
Coupling:      decoupled <──●───────> tangled   (don't go right)
Flexibility:   application <─────●──────> library
Cohesion:      convenience <───────●──────> orthogonality

Because the forces that pull the slider one way or the other are almost never about programming — they're about money and people: how many programmers you have, how many artists, who pays whose salary, and who's going to be cut to save costs. The architect, on the other hand, designs the system of systems, and couldn't care less about you, money, and people... provided, of course, he's a good systems architect.

Patterns aren't about the programming language

A small lyrical digression. "Pattern" is often translated into Russian as "template" (shablon), but a template is a rigid form that allows no variation, and that's the meaning that stuck in Russian. In English, several different words are used for the various descriptive parts and variations of development, which considerably enriches speech, the context of understanding, and conference talks themselves.

pattern doesn't fix a form; it describes forces and how they get resolved, leaving the concrete implementation open. GoF inherited the word but slightly gutted its meaning, and their patterns are much closer to templates.

template is a rigid form with holes for substitution. A C++ template would be the limiting case, where instantiation is deterministic, there are no variations, only along the type parameter.

idiom is like a low-level, language-specific convention ("RAII", "CRTP", "copy-swap"). An established turn of phrase in a particular language, and it's closest to the linguistic meaning of the word as a recognizable expression.

blueprint is a plan that must be carried out literally, i.e. an engineering drawing.

archetype is like a prototype from which variants are generated through deviation rather than substitution. archetype is closer to a "vision" than to a template, but in the other direction — it's more of a source of a decision than a network of similar decisions.

motif is used as a recurring element, recognizable but without any claim to "solving a problem". More common in music, but I've also come across it in technical documentation when describing systems.

schema is an abstract structure of data/relations, from the world of databases, and it's no longer about solving a problem but about the form of organizing data.

convention is an agreement, motivated by nothing except the accord of a few people ("naming convention").

paradigm is an order of magnitude higher in level of abstraction, a whole way of thinking (OOP, functional programming).

In the Russian community many of these words get replaced with the single word "pattern". It's too late to change the terminology, of course, but keep in mind that along with the word "pattern" you have to read the context in which it's used. When people say "design patterns", they usually think of GoF with their factories, singletons, and the rest of the menagerie — but almost everything there is design for change, meaning we're already inside the "how do we program this" picture of the world, and those patterns are more template, blueprint, and idiom than patterns proper.

But no matter how good the architect and designer are, guessing the scale of the coming catastrophe on the first try is nearly impossible, so most people start with what they lovingly call rapid(insert another word here)-coding, a.k.a. "coding like it's 1999", and then the fun begins — because despite the seemingly wide-open room for growth, there are only four paths engines take. In the life of any studio writing its own engine (let's call this Phase-0), there comes a moment of "hey, let's build a construction kit", and we move into Phase-1...

Everything is an object...

Here you usually have the hacked-together code of one or two games, where everything is done with classes. For everything to be done with classes, you need a GameObject, a.k.a. GodObject, that knows everything about everyone. And there's a second God, a.k.a. GameLevel, that gets passed around everywhere and also knows everything. Right now everything is connected to everything, and our GoblinLeader knows how the sun works in the sky, because the sun is also a GameObject.

class GoblinLeader : public Goblin {
  /* parameters + behavior */
};
// the hierarchy is roughly this:
// GoblinLeader -> Goblin -> Enemy -> AiObject -> MoveableObject -> GameObject

The programmer also knows everything about the goblin leader: who he is (baked into the inheritance), what he's like (field values), and what he does (method code), and that's good. But "who" and "what-he-does" are stuck together for good, because a class is data plus code in one bottle, and that's bad.

Phase-1 is very crooked, but it works, and it works for a long time. On the subject of the GodObject there's a wonderful tale from "Blitzkrieg" told by the developers themselves: at some point the neutral pigs inherited from the combat unit and, upon seeing an enemy, patriotically reached for a weapon they didn't have — but they did have ammo for it, thanks to an uninitialized variable, and since they had no weapon, they mass null-pointered and crashed the game. They fixed the crash by giving the pigs a Parabellum but taking away the ammo, i.e. initializing the variable to 0.

This tale captures the whole essence of Phase-1: the entire engine lives off inheritance from a single class, and bugs are cured with inheritance too. Next comes Phase-2, "things can't go on like this", but as I already said, it can be very far off in time. So far off that years can pass before the team finds the strength, time, and money to transition to Phase-2. Take a look at Wargaming with its more than ten years in production, a pile of games for every platform, weekly updates... and as they themselves admit, the engine is still in Phase-1, and everything's flying just fine.

... but not everything can be made an object

Finally a critical mass of "things can't go on like this" builds up, and a team is set aside that starts bringing order and beauty. Modules appear, managers, well-structured classes, dynamism shows up, and instead of rigid inheritance you start getting questions like "does this object have a component with such-and-such name?".

GameObject* player = scene.findGameObject("Player");

if (!player->hasComponent("ControlComponent"))
  player->addComponent(new ControlComponent());

player->update();

GameObject slims down to a container of components, code cohesion drops, system coupling grows. When cohesion drops that's always good, and what's bad is that in exchange you get an abundance of type casts, hand-rolled searching for sub-objects, smart pointers everywhere (because it's now impossible to tell a whole entity from a part of it), and messaging — a mechanism that lets you tell an object it's already dead, while how exactly it'll be dead and who deletes it is decided by some other system. And the worst trash, as usual, always settles in the GUI system; the phrase "it just ended up this way for historical reasons" is most of all about the GUI.

Phase-2 usually takes about a year or two, because it's done by programmers. A year, honestly, is very, very little to migrate a big project, and the shorter Phase-2 turns out to be, the better the code was before the cleanup. For example, at Unreal the migration took almost 7 years, from 2007, when they announced they'd be moving toward config-code, to 2014, when it all finally worked.

... but it can be made a config

And then the magic starts happening, magic the very programmers who wrote it are no longer able to control. If we already have a description of the object in data, why keep the GoblinLeader class in code at all? Remove it. Let the object be assembled from an external file.

// there's no more GoblinLeader class, there's a description

{ "GoblinLeader": {
    "EnemyComponent":    { /* ... */ },
    "MoveableComponent": { /* ... */ },
    "BossComponent":     { /* ... */ }
}}

// ^ what moved into the file is not just the constants, but the object's assembly recipe itself

This is usually when a truck full of rakes pulls up and it turns out the description of a simple little window has moved off into some XML, and now that XML is managed by Vanya-John-Vapor, who also needs to be paid, and for that money he'll be writing coordinates into XML. And Vanya-the-junior-programmer suddenly becomes a technical artist, while the whole beautiful OOP model of the application moves off into the meta-model of the window system, which is configured by other people, and the application, from the programmer's point of view, becomes alien and unfamiliar.

The programmer no longer knows "who", and there's no goblin leader in the code — there are his separate parts, and even those parts aren't complete, they're assembled from some other parts. And to the rendering code the leader is no different at all from a rock or a pig, and might quite happily be storing a Parabellum without ammo, or even ammo without a Parabellum separately, and with no desire to crash the game. On the plus side, the game designer assembles from the components things the programmer never even imagined, and fixes the pigs without the programmer — and sometimes faster than him.

A telltale sign you're approaching Phase-3 is the moment when you know the components individually, look at something in the game, and sincerely don't understand how the designers even managed to put it together. On the organizational side, at this moment the studio splits into departments, grows sprints, meetings, and zones of responsibility that nobody wants to take on because "that's in someone else's zone", and separate professions appear like interface layout specialist, monster tester, CI engineer, and technical designer for levels and locations.

And here, in this phase, programming and C++ code come to an end, and the flight of the designers' imagination begins — which is exactly what everyone loves Unreal for. But in exchange you get a heap of problems and questions about how to support that flight. How do you serialize? How do you debug logic that's moved off into data? How do you merge two conflicting level-settings files? All the questions that in the world of pure C++ had answers long ago now have to be solved from scratch.

And on top of that comes a duty that absolutely every programmer without exception hates — rewriting this logic back into C++ for the game designers when they've botched something, or it doesn't work for them, or it works hellishly slowly. And you end up with positions like technical designer, i.e. an elephant-fly hybrid, because it's impossible to spin this off into a separate specialty, and nobody will voluntarily go rewrite someone else's bad code (and someone else's code is always bad).

... through your own programming language

The circle closes: now all the game code has been moved out of the engine, the engine itself has turned into a virtual machine, the whole game has become just content, and all that's left is to give game designers the ability to create content themselves, without programmers involved.

ScriptEngine::ExecuteScript("GoblinLeader");
// "GoblinLeader" is a file describing both the parameters and the behavior logic, with 0.001% C++ code

The sign that this phase is beginning is recognizable when the company again tries to bring programmers and designers closer together, technical game designers appear, and programmers are asked to lay out the windows themselves. But programmers want to write scripts as text, ideally text that can do everything C++ can, plus what it can't, while game designers want little windows, sliders, and visual programming. The side that usually wins is whichever ideology has more people in the team: where there's a programmer culture, they do it through scripts; where there's more art, they want components.

... data-driven is inevitable

Absolutely not. This is probably the main thing worth taking away from the whole text: the very meaning of data-driven is the introduction of new, usually non-programmer specialties, which for a small team of nothing but programmers would simply be harmful. The programmers will spend a long time and a lot of money maintaining the Tower of Babel they've built, one that only pays off at scale.

If three programmers and code of the corresponding quality are enough to make a game profitable, then you'll have three programmers and code of the corresponding quality. And that's fine. The early days of the App Store, hypercasual games, the era when home PCs first appeared — all of these were examples where bringing in data-driven beauty was overkill. Besides, it's not 2006 anymore, and you can grab Unity, Unreal, or Godot and use them either as a data-driven construction kit or in the pure "let's just hack it out" style. And you can make a mess in the code just as well as in the data... that's actually the favorite pastime of beginner teams.

... to be continued

Twenty years ago smart people already described this path from objects to data and back to languages. You might think that by now everyone has long been writing "like in phase 4", but in reality living companies sit on all four phases at once, and that's fine, because the phase is dictated not by fashion — though by fashion too... (ECS has become exactly this kind of cargo cult) — but by money and team composition.

Unification of tools, shared engines, visual scripts: none of this has gone anywhere, but the magic of programming hasn't vanished from the world either, and programmers are still needed — game programmers, engine programmers, technical, UI, scripting, sound... and who-the-hell-knows-what-else programmers.

Gregory's wheel keeps turning, and following the desire to build the whole game in a visual language will come the desire to get performance, and after that the desire to better manage the data schema, and somewhere on this turn you'll again want your own DSL, or blueprint, or daslang, and it all inevitably starts over from the beginning.

Materials

P.S. So then, the promised explanation about the first paragraph.

The most expensive engine, the one it's easier to throw out than to understand and fix, is CryEngine — or rather its fork, on which Chris Roberts has been chipping away at Star Citizen for more than ten years, and hundreds of millions of dollars of crowdfunding went, not least, into rewriting the engine to fit their needs. Even the engine (the engine, not the game itself) of GTA VI is estimated by the industry at "only" $220M, almost twice as cheap as Star Citizen.

The studio that lived for ten years on sh...code is Mojang and the original Java codebase of Minecraft, which the community and the developers themselves jokingly called "you can't delete lines" — but that didn't stop the game from becoming the best-selling one in history, after Tetris of course :)

And the architecturally correct engine that died because the company ran out of money to develop it is Bitsquid, later renamed Autodesk Stingray, with a clean data-oriented architecture from the very start, one that seasoned game devs still write about in blogs with bated breath. It was shut down in 2018, because the market moved en masse to Unity and Unreal, and a perfect architecture with no real projects turned out to be of no use to anyone.

← All articles