While writing the source code of a 3D engine, we can have many types of problems. These may range from simple problems of code that will not compile, to harder problems like memory leaks. At IMVU, we have automated solutions in place to help find these kinds of problems. For example, unit testing combined with Valgrind has helped us track down a lot of memory leak problems.
But what about performance problems?
A single code commit can subtly affect the speed and performance of the 3D engine in ways that are not obvious to the engineer. We suffer from this problem enough that I found it worthwhile to look into in more detail. We hacked a rough solution trying to minimize the performance regressions we had in our 3D engine, and the solution quickly paid off, informing us which were the commits affecting the performance of our 3D engine. While it wasn’t part of the original IMVU schedule, we encourage engineers to solve problems they see, and I took it upon myself to create a tool that solved this problem. I spent several late evenings and even some weekends to build what we now call BlueSteel.
BlueSteel is a tool to help track the performance of our projects by commit, branch, and machine.
Given a set of commands that we will call “Benchmark Definition”, BlueSteel will execute them and will read and store its output on something called “Benchmark Execution”. In the case this output follows an specific schema, BlueSteel will understand the data and will inform the commit author if a fluctuation exists between Benchmark Executions. The ability to check every commit on any branch in our projects is very important.
This gives us the ability to quickly react to a performance degradation, and more importantly, the author of the commit, who has the most context on that code, will be notified very quickly, before that context is lost. In addition, it gives us the ability to freely experiment with new ideas on non stable branches and see whether the resulting code performs better or worse compared to the stable one.
As an example, we are going to expose a real situation of how BlueSteel detected a performance issue, and how BlueSteel helped us by notifying us to fix it. We started from the idea of splitting the list of elements to render between opaque and transparent ones. This was an experiment in preparation for rendering transparent elements using the Depth Peeling technique. With git, we created a branch to start the experiment, and we committed some code. After that, the commits were pushed to our Github repository. At this point, BlueSteel took few minutes to notice the change and feed all the information to its database.
git checkout -b wip-render-slot-split
git push -u origin wip-render-slot-split
Afterwards, we went to the BlueSteel main page, which allows us to manage the different concepts listed below:
- Layouts: A Layout is a group of projects that work together for benchmarking purposes, and this tab will show us the list of Layouts configured in BlueSteel.
- Projects: A project represents a git repository with the code we want to benchmark or the project we want to use to benchmark other projects.
- Definitions: A Benchmark Definition is a set of properties. These properties are:
- A list of commands that will give the result of the benchmark.
- A list of allowed machines to run this benchmark on.
- Fluctuation thresholds and overrides.
- Notification controls.
- Executions: A Benchmark Execution is the combination of Benchmark Definitions, commits, and machines. Each Benchmark Execution will store the result of a Benchmark Definition executed while the project was checked out at a given commit on a certain machine. When all these Benchmark Executions are stacked together, then we are able to see the progression of our performance on a branch.
As a quick note, we have a Benchmark Definition configured with the following commands:
Image 1
This Benchmark Definition is going to be executed on af050975 and benchmark3 machines:
Image 2
And BlueSteel will send notifications if it detects a fluctuation greater than 2%.
Image 3
If we click on the Projects tab (1), we will see a list of registered projects and each of these projects allow us to list all of its branches (2).
Image 4
After clicking on ‘See project’s branches’ button, BlueSteeI presents us with a paginated list of all the public branches of our project and a quick view of the status of every commit. In this page we can see that BlueSteel is aware of the branch we pushed, and all the required workers are already working on it.
Image 5
After that, we clicked on the ‘Executions’ tab, which allows us to retrieve results of Benchmark Definitions and commits within a machine. In this case, we selected:
Layout: northstar-layout.
Project: northstar-project.
Branch: wip-render-slots-split.
Definition: cpp-render-executer.
Worker: af050975.dmz.imvu.com
This combination of options gave us the following result:
Image 6
As we can see, this branch only has 3 commits on it (the most darkest blue ones), and the rest of the commits are from the master branch. Also, we can see that there is a problem with the latest commit. The time spent on the main render pipeline increased more than expected. The increase of time in this commit was greater than the previous result of the previous commit. The maximum allowed increase between commits, defined in this Benchmark Definition, is 2%.
We can have a look at the email that BlueSteel sent us because of this issue:
Image 7
We can see that the fluctuation value was ~2.17% compared to the previous commit, enough to bring our attention to the affected code and investigate what was causing the performance degradation. After a while, we discovered that splitting the list of render slots into two lists was forcing the engine to make more pointer de-references than expected inside a high frequency loop. Because of this information, we took a step back and rethought the whole strategy for the required solution.
Now the render pipeline has a more optimal way of separating opaque and transparent render slots without paying hardly any penalty on performance.
BlueSteel is now a publicly available tool under the MIT license: https://github.com/imvu/bluesteel
Llorenç Marti Garcia