DT: Performance++

Hey everyone, welcome to another Stonehearth Desktop Tuesday! We hear a lot from you that one of the main things that would make the game much better is improved performance. We totally agree–bad performance can cause everything from slow framerate, to the UI going unresponsive, to hearthlings stalling out on tasks and buildings and restocking forever. Let’s take a closer look at the things that cause bad performance, how you can diagnose it when you’re experiencing it, and what we’ve been doing to make it better.


Recap from the video:

First some performance 101: Stonehearth’s performance on your computer is mainly influenced by two major factors: your CPU, which is how fast your computer can compute instructions, and your graphics card, which determines how many things–people, lights, etc can be drawn to the screen at once. It also helps to have a lot of ram, or memory, because whenever the game is running, it needs to keep track of everything in the world, and a hard drive that’s an SSD, because that way, it’s super fast to read saves and other stuff that falls out of ram off of disk.

The most common and frequent performance problem that Stonehearth has is that’s is a CPU hog. Unlike most shooters or platformers, simulation games–here represented by the fact that every hearthling and enemy in Stonehearth has it’s own AI tracking it’s hunger, health, job requirements, tasks and more–generate a ton of work for the CPU to do, so much that if you have enough hearthlings in the world, the CPU cannot calculate all the things they need to do in time to make the hearthlings look intelligent. Since hearthlings are programmed to do simple things if they cannot figure out how to do complex things, CPU-related performance problems manifest as hearthlings going idle, restocking forever, or having conversations, since all these things are “default” actions that happen whenever the AI cannot figure out how to do complex but high priority action like combat or basic needs or building.

To fix CPU related problems, the engineers on our team run code profilers. For example, here are some screenshots from Telemetry, a radgames tool that lets you see which parts of the game are eating up the most CPU time. Functions within the game’s code appear on the left, sorted by the amount of time they take to execute, which appears on the right. To optimize C++ code specifically, we also use vTune. In this particular screenshot, you can see that the act of allocating memory, or using the malloc() function, is quite expensive relative to everything else in the function. We also have a homegrown tool that does a similar analysis for our Lua code, which is where the bulk of our game logic resides. We collect the worst offenders into a google spreadsheet, which acts as a hitlist of parts of the code that the engineers need to optimize.

Based on what the profiler has identified, engineers go into each expensive line of code and work to make it less so. For example, in the above case of an expensive memory allocation, it may be possible to re-write the code to use fewer objects, and thus require less memory. In other situations, it may require caching data so it does not need to be recalculated, or, in extreme cases, moving code from Lua, which is easy to develop in, to C++, which runs much faster.

Sometimes, optimizing for performance can mean fundamentally re-writing a section of the code so that it uses an entirely different mechanism than it did before. For example, when Engineer Max noticed that restocking was soaking a lot of CPU time, because hearthlings must do a lot of pathfinding to consider what to move and where, he took the restock logic out of the usual AI paradigm in which each worker calculates their own work, and moved it into a global restock director, which does one calculation for the whole game, and then assigns restock tasks to the entity that must do it. In this gif, you can see a bunch of hearthlings attempting to restock thousands of items the old way on the left, and their new efficiency, post-restock-director, on the right. On the down side, restock now works a little differently than all the other systems in the game, making it a bit harder to learn the code and mod or modify overall.

The reason performance work comes at the end, therefore, is often because the price of making this faster is that the code becomes less readable and harder to work with. Lua code, for example, allows a developer to make a change to the system without recompiling the game, which can take several minutes. Moving a Lua function to C++, as we did with our event system, our pathfinder, our AI core, and more, means that forever after, developing that part of the game becomes slower.

The other major source of performance lag is from your graphics card. Like your CPU, your graphics card is given thousands of commands to process each time it renders a frame to the screen. The more people, particles, lights, fog, transparency and shadows are in a scene, the harder your graphics card has to work to render everything. If there are more items on screen than your graphics card can handle, it will render fewer frames per second, and if the frames per second ever fall under 30, your game will appear to lag as you attempt to scroll around and monitor your village.

To optimize graphics performance, our team uses nVidia nSights, a tool that breaks each frame down into each piece required to create the final picture. You can scrub through a timeline and watch the whole view come together piece by piece–where entities are calculated and placed, where color is added, where lights are added, etc. In this particular case a lot of time is spent rendering lights and shadows, which is why the game tends to hitch at night and in towns with lots of lamps. To fix this Engineer Chris and Engineer Angelo altered how our lights were being created and how shadows worked. If you’re still having trouble after these updates, check out the graphics page on the settings screen, and turn down the settings till you have something that renders smoothly.

The last thing that significantly impacts say, multiplayer performance, is how data is sent between the client and the server, which is why it’s best if the host of a multiplayer game has a fast upload speed. If the data from the host gets to the client slowly, the client can also experience lag as their simulation waits for the host’s data to arrive. To fix this we do things like compress the packets before sending them, and sending only diffs–the data that has changed since the last packet sent.

And that’s an overview of the performance issues that Stonehearth encounters, and a brief glimpse at the many things that the engineering team is doing to address them. Unfortunately, due to the fact that our game only uses three threads on account of the difficulties involved in creating simulations across multiple threads, there’s only so much that can be added by playing the game on processors with multiple cores. That said, in the last few months, we feel like we’ve made quite a bit of progress, especially on end-game towns and saves. Engineer Max’s Emerald Overlook, for example, used to be unplayable at 30 hearthlings; it now purrs along quite nicely. As we continue to push betas, please continue to send us data on how these optimizations are going for you. In particular, if you have a computer at the recommended spec and a save that’s lagging though you have fewer than 40 hearthlings, please send it to us! Since everyone plays the game differently, it’s always possible that we’ve optimized a code path that we use that you do not, and we’d love to see what’s going on that is making your games slow.

Other Announcements

Next week, we’ll skip Desktop Tuesday on account of American 4th of July. Desktop Tuesday will therefore return the week after, on Tuesday, July 10th.

Thursday streams, however, which happen at 6:00pm PST on www.twitch.tv/stonehearth will continue as usual. See you there and see you in two weeks!