DT: Autotests++

Hey everyone, welcome to another Stonehearth Desktop Tuesday! Now that the new builder and multiplayer are both out in your hands, the team has pivoted to working hard on performance and bug fixing. What does this process look like? Let’s take a closer look!


Recap from the video:
As always, we comb the forums for your bug reports and saved games–we especially love saved games that show the bug happening in a reproducible fashion, and then we look into the code see what might be wrong. So thank you for all your reports, and please keep posting them!

In addition to that, Engineer Angelo and Engineer Justin recently added an automatic error error reporting system. This means that if you’ve opted in to send us your anonymous analytics data, we will also be notified of errors that pop up while you’re playing the game. These errors get aggregated into a service called sentry. From the incredibly useful Issues dashboard, the team can see the errors that have appeared most often, and how many unique players they affect. We can see their frequency on an hour-by-hour basis, and on a day-by-day basis, so for difficult things like AI errors, which can have multiple sources, we can check to see if releasing a fix really does cause the error to decrease in the wild, or if there are perhaps other sources for it.

However, because code is code and every system touches every other system, every single time we fix a bug, there’s a chance we’ve broken something else in some other part of the game. One of our best tools to make sure that this doesn’t happen is to create an automatic testing suite–autotests for short. Each autotest is a tiny version of the world that tests exactly one piece of functionality. For example, in order to test that farmers will farm fields designated by the player, this autotest creates an empty world with one farmer. We have commands that simulate clicking on the ui, dragging out a farm, and selecting a special kind of fast-growing test turnip. The autotest then checks to see if the farmer can harvest two baskets of turnips–automatically tilling, planting, harvesting and then replanting and reharvesting the space. If this condition is seen, the test succeeds. If the farmer gets stuck, the test fails.

Engineer Tony created the first autotests back in 2013, and they were the subject of one of our earliest Desktop Tuesdays. In the years since, we’ve run the autotests before every single code commit, and each member of the team has added their hairest functionality into the suite; the things they most want to make sure nobody else accidentally breaks. As a result, we spend a lot–and I mean a lot–of time, staring at autotests other people have written. As a bonus, we thought you might enjoy them too! Check out the video for an in-depth look at what’s going on per-test.

We know we’ve still got lots of bugs. While playing do you see anything that commonly breaks that should have an autotest attached to it? Please let us know!

Other Announcements

This week, Angelo is streaming at 6:00pm PST. Bring all your multiplayer or graphics-performance related questions. He’s using nVidia nSight to break down frames by their components, and it’s really cool.