TickTock Games recently ported the PS3 game BurnZombieBurn to Nvidia Shield. In this article Arden Aspinall explains how they did it.
Arden Aspinall is the CTO of TickTock Games. He specialises in 3D graphic development and code optimisation. With over 25 years’ experience, he and his team are constantly looking for new challenges that push forward the boundaries of mobile game development.
TickTock Games Limited
ECg - getting to the heart of the issue
When we started the port of BurnZombieBurn, we knew performance was going to be a massive issue. This was, after all, a game that was designed to look and play amazing on a PS3. However, we also had the small issue that Cg, which the PS3 game uses heavily, was just not available on Mobile devices.
With this in mind, before even attempting to port to the Android we created a PC reference build and ripped out the Cg libraries from the game's beating heart and replaced it with a thin Cg API implementation of our own, which redirected all the Cg calls to GLES compatible functions.
Of course, all the Cg shaders had to be refactored into GLSL. We originally thought this would be a complete nightmare; there were literally hundreds of shaders! Since we did not have the original source game assets we wrote code to extract the source from the game's data files and performed analysis on usage.
We quickly realised we could create a hash from the shader's source code and redirect this to our refactored GLSL code. Not only that but we could reduce the number of unique shaders to a very manageable 30.
ECg worked extremely well, and we very quickly resolved our bugs and our PC/GLSL reference build was running beyond our expectations.
Initial port to Android
Here at TickTock we are no strangers to Android development. Before embarking on this latest adventure we had already shipped six other Android games.
However, our partners at NVIDIA were keen that we should use their newly released TADP
(Tegra Android Developer Pack) with Visual Studio Integration. Up until this point we’d had to spend a considerable amount of time streamlining our Android development pipeline.
NVIDIA’s single install TADP ( Tegra Android Development Pack ) gives you everything you need for Android native development. 15 minutes of install time and you can start development out of the box, no matter if your preferred development environment is Eclipse or Visual Studio. I just wish it had existed when we started Android development. I can’t recommend TADP enough.
With TADP installed, it was just a case of adding a new platform to our PC reference build. As the game was built for the PS3, it would just compile on the Android GCC compilers, right? Well, no. GCC had moved on a bit and we had some very interesting compiler errors. However, being able to work in the familiar environment of Visual Studio meant we could quickly jump around the code, diagnose the issues and fix them.
After stitching in a bit of Java to get the basic Android lifecycle, it wasn't long before we could hit the build solution and spit out an APK ready to start testing on Android.
With great trepidation, we ran the game for the first time. Expectations were very low that anything would work so early on. We’d set the screen to clear with random colours, so if we saw flashing colours on the screen that in itself would be a huge step.
I’ve been at this point in the development process countless times, but what happened next was a genuine surprise. We had no flashing colours. Out of the blue, the music started playing. The only bit of middleware used in BZB is FMOD. If audio in game development isn’t your thing, this result really is testament to how good an API FMOD is.
Despite being a bit perplexed as to why we had no flashing colours, we decided to take things one step further and redirected a screen touch to act as the fire/select button. To our amazement, we found we could get right into the game. We started hearing the groans of the undead and the screams of our invisible hero Bruce as the zombies attacked. Probably a good reason to be scared of the dark.
We felt it was only fair for Bruce to get the game rendering. To do so we took advantage of PerfHUD ES
, the tool that gives you complete control of development. As any graphics developer will know, getting things rendering at the beginning of a new project where you have no existing code to draw upon is always tough. Doing so on mobile platforms even more so.
PerfHUD ES, takes all the GL Commands from your mobile device and reconstructs them in its own Windows App. You get performance statistics, the ability to see stage by stage how your scene is built, from the comfort of your own PC. Throughout the development of BZB, we began to see the power of this tool and discovered really cool ways to use the information it provided.
So through looking at the GL callstack and examining our render states, both features of PerfHUD ES, we quickly worked out we needed to modify the way we attached the Z Buffer to the render target and add a little extra code to present the screen.
It wasn't long before we had our parity Android build running at a staggering 0.5fps. It looked amazing. It sounded amazing. The trouble was, it didn’t take a super hero to work out it was completely unplayable!
It was a long, hard road getting BZB running at a decent frame rate. When we initially started talking to people about this project, most people thought it was impossible and crazy. A few people at NVIDIA believed that it was possible, but even they knew it would be hard.
Very quickly we worked out we were CPU locked. Unfortunately for us, the Performance Analyser that is now part of the TDK wasn’t available. Fortunately, we had our PC parity build, so to understand where our CPU cycles were going we profiled that. It wasn’t perfect but it gave us a good view on where best to spend our time.
The biggest bottleneck was somewhat of a surprise. It turned out it was taking a considerable amount of time sorting the game's geometry into depth order. Clearly, on the PS3/PC, this wasn’t an issue, so we quickly implemented a few performance counters on the Android version and found the process was taking a minimum of 80ms per frame!
The camera system in Burn Zombie Burn is always above the action, so this was a fantastic opportunity to pre-sort the geometry on load time and cut that whole process out.
With this issue out of the way, we discovered the next CPU bottleneck was the animation and skinning. We’d been advised by NVIDIA that it was probably best to keep the skinning on the CPU, so we set about optimising the old fashion way. First, we wrote code to find the most significant bone blending influence so we could reduce the number of bone influences per vertex from 4 to 1. I can imagine any graphic designers cringing at the thought of this and to be honest our original plan was to implement this as a level of detail. However, we abandoned this idea and stuck with the original optimisation as even the most critical eye couldn’t see a difference.
The game was now running between 6 to 9 fps and things were starting to get exciting. The game was playable; now it was just a case of getting the frame rate higher to get the responsiveness and smoothness we wanted.
At the time, we were targeting the Tegra3 chipset, so we knew we wouldn’t have the bandwidth to do full screen post process effects. This was something that Burn Zombie Burn needed so we had to come up with alternative solutions for things like fire. Although it didn’t quite look the same, we tried to get it looking as similar as possible.
Although we had started doing some heavy duty optimisations to the game, we made sure we could switch these on and off, so we could drop back to the Android Parity build at any time. As a team there were a LOT of arguments about this, but it was the best thing we could have done. It meant that as we found bugs in the game, we could quickly work out if it was due to our optimisation or due to an assumption made during the original porting process.
We’d now got the game up to 12fps; still not fast enough, but the momentum was finally on our side.
It was time to fire up PerfHUD ES again to see what was going on. We started looking at the Polygon count; our brief was to use the original PS3 assets, so reducing them wasn’t really on the agenda. However, they did seem ridiculously high. It turned out that all the geometry was rendered with indexed tri-strips, and the extra triangles were the degenerates. Fixing this was simple; on load time we just converted them to index tri-lists. The tri count was literally halved and the frame rate increased slightly.
PerfHUD ES also showed us that we needed to get our geometry into VBOs. Fortunately, the original game engine did allow us to implement this relatively easily. However it took slightly more work than we originally planned as each section of geometry created a different VBO for each type of element within the vertex buffer. Ideally we needed them interleaving.
Being able to see how the textures were rendered allowed us to rule out issues there. Tegra chipsets support DXT1/5 texture compression which is what was embedded into the game assets.
The ability to look at shader usage from within PerfHUD ES gave us an amazing insight for optimisations. The ability to edit them on the fly and upload them back to device was really useful. Initially, it helped to get rid of the issues we introduced when we first refactored the code from Cg to GLSL. Later, it helped us optimise our shaders, moving things from the fragment shader to the vertex shaders to help improve performance whilst ensuring the changes didn’t spoil the look of the game.
As we reached the 25fps mark, we finally started to see the light at the end of a tunnel, until one October morning when a mysterious package arrived.
Shields up; it’s time for a Zombie BBQ
The contents of the package were several circuit boards and a clear plastic housing. No instructions; this was clearly something cool, new and hot off the press from NVIDIA.
We put our new and mysterious device together to find it was Android based so we did what any developer would do, we ran our game on it. To say we were blown away was an understatement. With no changes from our Tegra3 version BZB was running at 60fps, never dropping a frame. Very exciting!
Our friends from NVIDIA asked us if we could produce a demo of the game which would show off what this hardware could do.
It was time to look at our optimisation switches. Surely this hardware wasn’t capable of all the post process effects?
Enabling them was simple, we hit the switch and although the frame-rate dropped significantly, the game was still running at just over 20fps. This was with all the PS3 post process effects enabled. We spent a considerable amount of time on optimising the shaders and talking with NVIDIA’s tech team on how we could achieve the same effects using less GPU power. In the end, the game was running at a respectable 40fps.
Our reward was to find out that BZB was going to be shown to the press at the CES when the Tegra4 was announced. The long days and the lost weekends were starting to feel like a price worth paying.
Looking back at the project, handling the Android Lifecycle was something we should have dealt with a bit sooner. We left it until the end of the project. We needed to prove it would fit in memory, run fast enough and be playable from a touch interface. We've done other Android games in the past and we believed we knew how to deal with the Android lifecycle very well.
However, this was not the best of ideas. Getting BZB working with the Android lifecycle was a real challenge. The assets with the game were compressed, so having these assets in an APK lead to abysmally slow load times (we are talking minutes).
We had to then restore the GL context, which meant restoring the VBOs, FBOs and Textures.
A nice little trick we learnt when dealing with restoring the GL context was to use PerfHUD ES to save the GL state to a text file before locking the device. Once the device is locked we allowed the game to restore and use PerfHUD to again save the GL state to a new text file. We then simply used a text comparison tool to see what the difference was. This process must have saved us days of work, as we quickly figured out where the issues resided.
A nasty issue that we had significant issues fixing was with OpenSLES. Previously, for all things sound related, we’d gone down the Java route but FMOD gave us the choice between this and OpenSLES. The Java route gave us a performance hit. However, we found out later OpenSLES has caught a lot of developers off guard with lifecycle issues. In the end, we had to make quite radical changes, shutting down the whole audio system when the game was paused and reloading everything when the game was restored.
For those who have never played BZB, the game is huge! During the development of the Android version, we played this game a LOT. Even now when playing we see little things in the animations, features of the vast array of weapons and game elements we’d not appreciated. The game is so rich and a true testament to the team who produced the original PS3 game. It really is a work of love.
NVIDIA were very supportive during development, providing us with great hardware and amazing tech support. Their tools and team allowed us to spend less time finding the problems and more time to concentrate on solving them. Their engineers were fantastically helpful and stopped us from taking dead ends when exploring ways to bring this product to market.
I don’t think many people would argue; bringing the PS3 version of Burn Zombie Burn to Android was always going to be a tough project. I’m very proud of what our team learnt and achieved during the development of this game. Hopefully, for those who have just started dipping their toes into Android development this article will give you some ideas on how to maximise the platform. Good luck.