Why game devs stay on C++17

For the last couple of years at conferences I've been hearing more and more complaints from people I know in gamedev that the current direction of "modern C++" doesn't match the needs of game development. The real, useful additions effectively ended with C++17, and attempts to adopt C++20 often end with the discovery of a pile of "Heisenbugs" and a significant performance drop — critical for us at 10–15% from build to build. Having knocked around various game studios — geez, it'll soon be 15 years that I've been here — I do have a little something to tell you.

All modern studios bigger than two and a half ditch-diggers, writing games in C++, C# or something close to it, use Visual Studio or migrate from their own contraptions to Unreal/Unity — which is also C++, by the way, albeit with its quirks. Historically it just happened that Windows and Microsoft were, are, and for the foreseeable future of the next ten years will remain the largest market for PC-and-console games, while the consoles themselves long ago became "well, basically PCs," but to avoid losing exclusives (and shekels) the vendors will never admit it.

Mobile is sort of separate, with its own pokémon — Mac and Android — but 95% of games are created, debugged and optimized in Visual Studio in one form or another, the rest is rounding error. Since the start of the golden age of gamedev (somewhere in the late 90s), most games were written assuming they'd ship on PC, and PC means Windows. And the heritage of many A+ studios is tied to Microsoft one way or another, even for non-Microsoft consoles and mobile.

Right now Visual Studio is the best C++ debugger in the world, with an unmatched ability to bolt on, parse and display practically anything you can think of, and if Studio isn't enough you can open WinDbg, which even lets you program it. Debugging is, in principle, what many people use Studio for, and they're willing to put up with the compiler's antics, weak optimization, a buggy STL and other bugs.

Microsoft recently bolted on a time-travel debugger and generally gives you tons of options for everything C++-related — if you work on Windows, of course — from custom symbol servers to distributed builds and compiler scripts, all the way to the fact that PlayStation builds (a FreeBSD-based system, by the way) can't be built outside the Windows SDK. I mean, there's a Linux SDK, but it's not quite functional — you have to dance with a tambourine like in the good old days, and it's not a given that it'll even work, while questions in the Linux thread on the forum sit unanswered by support for weeks. With Nintendo (a musl system) it's the same deal: their SDKs are always fresh, but good luck building a Switch build from Linux. How's that level of trolling? And when you're used to having all this at hand, switching from skis to crutches and a hand saw (debugging in the console) instead of the "Druzhba" chainsaw (Studio's debugger) is something you really don't want to do.

C++17 was picked up by developers almost without issues, but with the revolution of C++20 and later standards something just isn't working out. What I really like in C++11 and its descendants is the build speed and the strong typing. Strong typing is certainly a feature of C++'s recent evolution, when we saw a huge expansion of the traits system, things like nullptr and scoped enums to fight the legacy of pre-Columbian-era code — but they threw in auto as a bonus. That's cool and all, but auto slows down the build.

void vector<_TYPE_>::preallocate(const size_t count) {
  if (count> m_capacity) {                               // 0.000032s
    _TYPE_ * const data = static_cast<_TYPE_*>
                       (malloc(sizeof(_TYPE_) * count)); // 0.000087s
    const size_t end = m_size;                           // 0.000023s
    m_size = std::min(m_size, count);                    // 0.000342s
    for (size_t i = 0; i < count; i++) {                 // 0.000038s
       new (&data[i]) _TYPE_(std::move(m_data[i]));      // 0.000135s
    }

And here's how, with auto, the compile time of individual expressions that involve automatic type deduction goes up by three — almost three — times. Do you have a lot of auto in your project? I just put auto in place of the types.

void vector<_TYPE_>::preallocate(const size_t count) {
  if (count> m_capacity) {                               // 0.000030s
    auto data = static_cast<_TYPE_*>
                       (malloc(sizeof(_TYPE_) * count)); // 0.000195s <<<<<<<<<<<
    const auto end = m_size;                             // 0.000062s <<<<<<<<<<<
    m_size = std::min(m_size, count);                    // 0.000452s <<<<<<<<<<<
    for (size_t i = 0; i < count; i++) {                 // 0.000042s
       new (&data[i]) _TYPE_(std::move(m_data[i]));      // 0.000258s <<<<<<<<<<<
    }

And here's where I start to really dislike the people who like the almost-always-auto style. No matter how well-justified the use of auto is, it's simply slower, and it also affects the compile time of the expressions it appears in — and not for the better.

But once you turn on C++20, everything gets even worse. Those examples were under C++14/17, where the time fluctuated within a 2-3% margin, and now look at this. Just out of nowhere, +10/15% to compile the very same code.

void vector<_TYPE_>::preallocate(const size_t count) {
  if (count> m_capacity) {                               // 0.000038s <<<<<<<<<
    auto data = static_cast<_TYPE_*>
                       (malloc(sizeof(_TYPE_) * count)); // 0.000223s <<<<<<<<<<<
    const auto end = m_size;                             // 0.000074s <<<<<<<<<<<
    m_size = std::min(m_size, count);                    // 0.000509s <<<<<<<<<<<
    for (size_t i = 0; i < count; i++) {                 // 0.000054s <<<<<<<<<<<
       new (&data[i]) _TYPE_(std::move(m_data[i]));      // 0.000302s <<<<<<<<<<<
    }

https://godbolt.org/z/TMEbxEGG6 (C++14)

Top 3 (top-level only):
        Z:\compilers\msvc\14.41.33923-14.41.33923.0\include\vector: 0.175865s
        Z:\compilers\msvc\14.41.33923-14.41.33923.0\include\cstdio: 0.040779s
        Z:\compilers\msvc\14.41.33923-14.41.33923.0\include\cstdint: 0.013023s

https://godbolt.org/z/q5nva67qo (C++20)

    Top 3 (top-level only):
        Z:\compilers\msvc\14.41.33923-14.41.33923.0\include\vector: 0.250358s
        Z:\compilers\msvc\14.41.33923-14.41.33923.0\include\cstdio: 0.057683s
        Z:\compilers\msvc\14.41.33923-14.41.33923.0\include\cstdint: 0.018684s

As for strong typing, it's not all bad there. It's slower than plain types, of course, but it has taken root nicely and is being actively adopted by most studios that care about improving the quality of their C++ usage; and if you're willing to give up a bit of build speed, then strong types are your friend.

You can ignore compile time if you work at a non-game company with good internal build-farm infrastructure and potentially infinite compute power to compile any code you can write with your three-story templates. But big companies don't have it easy either: their ten-story templates, code generation and reflection eat up so much that compile time starts turning into real bucks on hardware — hence modules — yes, I might say something heretical here, but modules and all the fiddling around them were needed by the big shops, and they pushed it into the standard.

For gamedev, individual studios and solo heroes have zero use for it: an indie simply won't have a build farm, and everything is built either on GitHub Actions, or the project has grown and there's something like IncrediBuild and a little farm of its own — well, those 15-20 minutes of build time get fixed with various hacks and the lead's competent hands. Modules in their current state slow down or don't change the build time of a big project, if they work at all, but they require new people to support them. A game developer hears about modules either at conferences, or from "module salesmen" at mid-level interviews.

You could argue a bit about the relative cost of compile time and hardware versus the cost of a programmer's time, and I can tell you that hardware is the cheapest resource you can buy with money, but... there's always yet another BUT.

But hardware is real money you have to ask the bosses for, while hiring time and development time get smeared out over some later period. And it's hard for many gray-haired girl-accountants to make that call, because you have to account for the slaughtered raccoons here and now, while the code, the devops and the game — those come later, in a month, or even at the very end of the tax period. And company management almost always sees only the optimization of current profit. That is, when you come to management and ask for money to make the game build slower, they look at you with a smile.

The next expense after hardware is tools — you need a lot of them, as many tools as possible: warnings, static analysis, sanitizers, dynamic analysis tools, profilers and so on. Everything on the market, from Cppcheck to PVS and SonarCube, gets used where possible, because a bug found at compile time or in tests is saved time that you can spend on development. But here's the trouble: all these tools work well on non-Microsoft platforms, Linux and various Macs — which, as you understand, is very atypical for game development.

What do tools have to do with modern C++? One way or another, all these tools work well with standard and more-or-less modern C++; they get updated for the new features and capabilities of the language, otherwise nobody would buy them, while old standards are either just kept supported so they don't break, or abandoned entirely. So when you come and ask for money for new tools that are supposed to support new features that aren't being used, they look at you with a smile.

But fine, open-source tools — you can still tidy those up and get them running on Windows, and with the ones whose authors want money you can negotiate a port — but most of these tools are oriented toward standard C++, and... ours isn't standard. There's ready-made support for std::vector, but I don't use it, because I have my own static_vector/doble_ended_vector/small_vector/buffered_vector/hybrid_vector/onstack_vector living there, with all the bells and whistles, a custom allocator and equally custom iterators, and these tools can't check it. You can't blame them for that — they're also made to earn shekels, and bloody enterprise pays far more than all the AAA studios combined.

And so it turns out that most standard tools simply don't do what's expected of them. And if they don't do what they were made for, can you trust them? This distrust has somehow lived in gamedev historically — first we didn't trust frameworks and libraries, and before us the old-timers didn't trust holy C, so they wrote in assembly, but it's good those times are gone. Somewhere near the end of the 90s MS VC6 came out and people slowly began to trust C compilers, but C++ was still a sketchy contraption, so code was written in C, in a C++ style, sometimes even with multi-line comments.

By 2000 the game industry had figured C++ out; it was the golden time of design patterns, the first engines and big research in gamedev — everyone fancied himself a Carmack or a Sweeney and brought goodness in the form of custom STLs, while on consoles there was no STL whatsoever and everyone dragged in whatever they could. And consoles back then were the top priority and the main means of making money. In 2015 I still caught the use of GCC 4.3.5 (which is, what, around 2007 release-wise) for certification builds at Nintendo — that is, along with the main build you'd send another binary that had to be built with the old GNU, and the two of them passed certification together, and only a year later did they switch to clang, 3.5 IIRC. The key word is had to, because a lot of stuff wouldn't build with version four, or built with surprises.

And at the end of the 2000s two revolutions started at once: huge legacy codebases spilled out onto GitHub, and the problems with class hierarchies led to the development of the component-oriented approach in games and engines, which keeps evolving even today — that's what gave rise to the Entity-Component-System (ECS), and who knows what it'll turn into next. And is there even a single tool that can check and test component and ECS systems in games, or outside games? So when you come to the bosses and ask for money for a new tool that's supposed to test something or other, they look at you with a smile.

The second revolution was the active consolidation of engines, which stopped being a balalaika that only the development team — a ten-armed Shiva — knew how to play, and the community got the chance to influence development. Well, you understand that a million contributors is, indeed, a million contributors.

Against the backdrop of all these changes, the platforms for game development changed too. Platforms became truly multitasking — symmetric and asymmetric. Developers used to Intel had to adapt to custom hardware with heterogeneous CPUs (PS2/3), then to PowerPC (Xbox 360), then to even more heterogeneous architectures (PS3), then came the zoo of mobile devices where everyone is on their own. And with each new generation of platforms the requirements for CPU, memory and storage performance changed. If you wanted optimal code, you had to rewrite it over and over — sorry, there's no time to change the standard. And that's without even mentioning the internet's impact on games.

So historically it just happened that everyone had their own STL implementation; it's no secret that the regular STL containers aren't all that well suited for games. If I have to choose between std::string and char[256], I'll pick the latter, because it doesn't allocate memory, and I'll somehow beat the bugs with it, whereas many people still don't know what to do about memory allocation in std::vector.

All — well, almost all — STL containers have problems with controlling memory allocation and initialization in one form or another, and in games it's important to bound memory for various tasks; amortized O(1) time isn't good enough — right now it's nominally 1ms, but three frames later it's 10ms, so on average it'll be 1ms, but the freeze already happened and players are unhappy with the bungling code-monkeys.

Memory allocation is, in general, one of the most expensive operations, and nobody wants to drop a frame because of an unexpected alloc. The same goes for external dependencies — you always need to understand where the CPU cycles go, where and when memory is used, why a stall happened on a resource access. But until recently our beloved VS changed its ABI like gloves, with every update, and if you had lots of dependencies, a compiler update would hang Studio for a couple of days, leading to a rebuild of half the SDK and unexpected bugs in old library versions. That's why studios preferred small, easily integrable libraries, ideally built in plain C, that do one thing but do it well — even better if the library is open source, with a free license and no mandatory attribution. So when you come and ask for money for a closed-source library with paid support, they look at you with a smile.

And add to that the C++ game developer syndrome (Not-Invented-Here). You know why gamedev loves reinventing the wheel so much? The rakes lie in known spots for years — the main thing is not to step on them too hard on the turns.

Gamedev started with rockstars, with developers who worked on hardware that would come out in a year or two — they simply had no other option but to make their own, here and now. And game development is one of the few branches of development that has credits, where even a rank-and-file programmer gets listed in the game's credits, and the more of your own ideas you implemented, the higher your name will be on that list — after the names of the directors, producers, head salesmen, and the names of their wives, kids and dogs, of course. Writing code from scratch, out of your head, until your brain crackles — that's not only fun, it often advances your career and raises the number of nominal raccoons at the end of the month!

Boost is also quite the reinvented wheel, but it's not much loved here. There was one funny case when a studio hired an outside person for a lead position — a good, seasoned enterprise guy. He brought along a few of his own people and sold management on Boost to solve a specific problem. There was a near milestone, nobody could stop him, and, as they say, the stars aligned. Boost drove straight into the engine's main branch.

It worked — not that everything was smooth, there were difficulties with updates, but nothing critical. A year later the guy quit, but the library had already spread like an octopus across the whole project, and then the studio was bought and the (you-know-what) time came to merge into another main branch, but the presence of third-party libraries, including such respected ones, turned out to be unacceptable. At first we did sneak in a few really simple headers, but that was it. We had to uproot Boost and write our own — somewhere swiping its separate parts, somewhere rewriting from scratch. Later we repeatedly had to tweak the code of pseudo-Boost and its derivatives, but taking something ready-made or updating the sources to newer versions became ever more problematic. Now the code was ours, and we ourselves had to maintain it. So in places it stayed at Boost 0.9, even though GitHub already had 1.2 and 1.3. So when you come and ask for money for people who could pull a new free open-source framework into the engine, everyone remembers Boost and looks at you with a smile.

But first and foremost, Boost didn't fit because it was too bulky, tried to do too much and critically increased the build time — x4 without it — and since the team we were merging into had already been through all of this, the answer was a categorical no. A year of dragging it in, half a year of weeding and bugs — no thanks, I wouldn't want that kind of happiness even for free.

And although there are many successful projects that use it, the phantom pains from that failure haunt me to this day, and seeing a folder with Boost in a project, I mentally prepare for half-hour builds and walls of template logs about how I forgot to specify a pointer somewhere. Examples of successful STL and Boost usage in particular projects change nothing — such is the psychology.

I'm not the only one; for these same reasons many game studios develop their own libraries that replace the STL and offer specialized solutions. But you have to do it without going overboard here too: looking for an alternative to std::map or std::vector with small-buffer support is quite reasonable, writing your own allocators and containers — well, who's going to stop you, but writing your own analogs of std::algorithm or std::type_traits with no practical benefit is already a questionable decision.

It's a shame that STL is primarily associated with containers. They're usually taught first, so when people say "STL," std::vector comes to mind, when they should be thinking of std::find_if. That's what really should be taught in courses — the ability to use algorithms; anyone can write a vector.

What's even worse than custom STLs is the actual absence of automated testing. Why? Because nobody needs it, because correctness isn't all that important and there's no clear spec. It'll do, as long as it's fun. That's the whole essence of game testing. When I first got into development, I was very quickly disabused of the idea that it's even worth striving for realistic simulation.

Games are always tricks, a whisper behind the curtain, T-poses behind the back, simplifications and deception. Nobody cares about simulation accuracy, and if you have no clear spec beyond "it should be freaking awesome," there's essentially nothing to test. Roughly speaking, games aren't like other areas of C++ where a lack of correctness and crashes can threaten someone's safety or money. Maybe it does threaten it, of course, but it's your own fault — you bought everything in the first week of sales, so now wait for the patches of the first year.

Games taught publishers that code doesn't live long. What matters is that the game doesn't lag too much at release — that's a vendor requirement; we still have to test it, but automation isn't mandatory. To management, testing is a waste of time and money that requires experienced engineers who could otherwise be writing code or designing levels, but whose results are almost invisible. Why write a test that the level didn't break after the hundred-thousandth launch?

That time is better spent creating new features, content and music. In the short term it's much cheaper to use hired QA units from India, whom you can rent by the couple thousand heads for the monthly salary of a senior C++ dev, and who will play the builds nonstop for a day, a week, a month... So when you come and ask for money for QA, they look at you with a smile.

Besides the testing problem, there's another one — the scaling of debugging, or rather its non-scalability — it's like the joke about nine women and the theoretical possibility of getting a baby in a month.

The main problem with debugging is that it doesn't scale. You can't take ten programmers and fix a bug in T/10 time; in the ideal case you'll get T/1.5 with pair work, and beyond that the total real bug-fix time will only grow.

And that's all because the bug reaches the programmer as dumps or repro steps, and what do we do — we launch the debugger and set breakpoints. Sure, setting breakpoints can help find some obvious problems like crashes, but what do we actually do about the real bugs, the ones that remain after we've fixed everything else? The ones that happen only under network load, on low memory, on data races, or with some small, not-yet-identified group out of millions of players, or on faulty RAM sticks, in an Estonian build, at 3 a.m. on a Monday?

We do nothing, because we know nothing about them — and here QA comes to the rescue, narrowing the error's scope as much as possible, saving the programmer's time by processing fairly large data sets, without knowledge of the code, without sources, trying to isolate the problem, make it happen more often, looking at logs and analyzing carefully. And then they send it to me, and what do I do — launch the debugger or meditate over the logs.

And I have to dive deep into "modern C++," set breakpoints on the specific data I'm interested in, but the debugger is exactly the same as always. The C++, though, is new, bringing more and more optimizations, doing copy elimination for temporary objects, moving little numbers through registers instead of the stack, even if I didn't ask for it.

It doesn't affect the ability to debug, it's just that the compiler less and less often generates the code I expect to see. And the standards after the seventeenth change the code I wrote more and more. Maybe at some point nothing of my code will be left there at all, and there'll be yet another layer of bytecode (there were proposals in clang to lift parts of the IR higher up and do dynamic dispatch for different CPU instruction sets) that adapts to the processor on the fly. So when you come and ask for money and time for a new standard that spends more of your time on debugging, they look at you with a smile.

Then you go back to your desk and keep using C++17. You don't have to adopt new features you don't like. Almost everything you do now will keep being supported — I'm sure, for the next two console generations, which is 7-10 years. You'll still benefit from compiler improvements in the future; that's a normal strategy.

At the very least I know a dozen projects and engines, fairly large ones too, like the one I work at now, that still use C++98 with a small sprinkle of lambdas, tuples and coroutines, ignoring everything else. And I have to say — C++98 is still a great language for writing games, often winning on build speed and release-mode performance. But at some point you'll have to face changes, because you'll need to hire other people. More and more often these will be C++ engineers who know only "modern C++." And on one of those Mondays a generational shift of developers will happen, the way it did with assembly, C, C++98, C++11. Freezing functionality and postponing compiler updates won't work forever. Or will it?