[slideshare id=7322096&doc=html5-newuilibraryforgameschadaustinannotated-110320042423-phpapp01&type=d]
HTML5: The New UI Library for Games
At GDC 2011, on behalf of IMVU Engineering, I presented our thesis that HTML is the new UI library for developing desktop applications and games.
HTML is Winning
The browser wars are hotter than ever. We have four major browser vendors, each with significant market shares, competing in earnest. Features such as SVG, Canvas, GPU acceleration, FAST JavaScript engines, and even WebGL are widely available.
It’s a great time to be a web developer!
Some Terminology
When I refer to HTML, I’m really referring to the entire stack of web technologies: markup, style sheets, scripts, canvas, websockets, etc. Also, Mozilla’s branding is confusing: Firefox the browser is built by Mozilla the company and is powered by Gecko/XULRunner the technology. Internally, we use these terms interchangeably.
History of IMVU’s UI
2004-2007: Win32, OpenGL, and C++
In 2004, when the company was started, IMVU’s entire UI was written in Win32, OpenGL, and C++. This is a fairly traditional, though old school, way to build UIs. However, our framework was extremely rigid, and it was hard to hire people to work on it. We tell legendary stories of “I just wanted to move that button from here to there, but I couldn’t figure it out in a few hours, so I gave up”.
In addition, our iteration speeds were terrible. You had to recompile, maybe add some new functionality to our UI framework, and then restart the client. This made UI development hard enough that we simply didn’t do it. The product didn’t change much for 3 years.
2007-2009: Flash
In 2007, a couple engineers realized we had a problem and spent a weekend integrating Adobe Flash. Flash is widely available and, if you get permission from Adobe, you can distribute Flash with your application. We implemented support for Flash UI panels and rendering Flash to translucent overlays atop the 3D scene.
Using Flash to develop UI was definitely a huge improvement. We used the Flex UI library and its standard components to get a head start. We were suddenly able to iterate on our UI, and we could add capabilities like animation and video to our client.
That said, Flash wasn’t all roses. It consumed a great deal of memory, especially address space, and mxmlc compiles were sloooow. Finally, even though Flex is open source, we found several bugs.
2009+: HTML
In 2009, we began a project to rebuild our entire experience from first principles. Our very talented design team spent months iterating with paper prototypes and user tests, and produced a very snazzy client architecture. At that point, the engineering team had some implementation choices:
1) Should we build this UI with a custom framework?
2) Should we use our existing Flash technology?
3) Or should we try integrating a browser and use HTML?
We decided to give a browser a shot, and we started with Firefox, since a couple of our people had familiarity with the Gecko codebase.
Within days we had something up and running, and the performance was amazing! It loaded quickly, was super-responsive, and used hardly any memory. We instantly decided to use HTML for the entire project.
The UI we built in HTML matched the intended design almost to the pixel. Iteration was nearly live: and we could even do our development in Firefox the browser with the Firebug add-on.
We patched Gecko to support rendering documents into textures so that we could overlay them on top of the 3D scene.
This let us build sophisticated UI elements across our entire product.
Benefits of HTML
Lingua Franca
IMVU is a Silicon Valley company, and we hire people from all backgrounds. We don’t necessarily hire people for the skills they already have; instead, we assume they’ll be able to learn whatever we need. That said, it’s valuable for people to have some background knowledge, and almost everyone these days knows a bit of the web stack. You can’t expect expertise in the nuances of the box model, but everyone knows a bit of HTML and JavaScript.
Amazing Tools
The web stack brings with it a plethora of tools like Firebug and Firebug Lite. We use these tools every day to rapidly iterate on our markup and style sheets. We can even integrate Firebug Lite into our downloadable client and work with live data.
Ecosystem of Libraries
In the last few years, we’ve seen a proliferation of amazing JavaScript libraries such as jQuery and YUI. If you build your UI with web technologies, you can leverage these libraries directly.
Advertising
When we integrated web technologies into our application, we discovered an unintended benefit. We could now directly benefit from the tens-of-billion dollar Internet advertising industry, which has built an entire advertising pipeline around HTML, JavaScript, and Flash. One engineer integrated ads into our product in less than one week.
Demo
At this point in the presentation, I gave two mini demonstrations. First, I showed what Firebug Lite was capable of: JavaScript commands, real-time editing of attributes and styles.
Then I demonstrated the process by which we iterate on our UI by, in 5 minutes, adding a jQuery accordion to an HTML overlay and demonstrating that the change had already taken effect.
Performance?
The biggest concern we hear from others is “Wouldn’t the web stack be slow and take a lot of RAM?” There was a time when browsers were comparatively slow and heavy, but with today’s fast CPUs and the continual pressure to make browsers faster (rich web experiences, tabbed browsing), we’ve found that HTML has never been a bottleneck for us.
Gecko consumes less than 1 MB of RAM for each loaded document, and we even think we’d be able to make every seat node in the scene into an HTML overlay.
You do need to be careful with DOM structures with huge numbers of children. Many DOM operations are linear in the number of children or siblings, so you’ll want to structure your DOM with a b-tree.
You also need to be careful with image tag and canvas memory usage. Avoid allocating for elements that aren’t immediately visible.
Drawbacks
Flash is still better at animation, but I think this will change. This may simply be a tool-chain issue.
3D on the web is possible now, but WebGL isn’t quite prime time. This year we expect WebGL penetration to reach 50% of the market (Chrome and Firefox), but we still depend on our software renderer for our customers without strong or reliable GPUs.
Tracing JITs are still RAM-hungry and don’t match native code performance yet. Core game components will still be written in close-to-the-metal C++.
Who else uses HTML for UI?
The independent game studio Wolfire uses WebKit for an in-game material and shader editor. They used the off-the-shelf editor component, CodeMirror, for their text editors.
Electronic Arts ported WebKit to PlayStation 3 for Skate 3. About 50% of their UI was written in HTML and JavaScript.
Netflix on PlayStation 3 uses WebKit so they can split-test their UI after they’ve shipped. That way they can improve their UI without shipping new software.
Several MMOs have added browsers so they can include forums, wiki, and help within their games.
How do I get started?
There are several open source and commercial libraries to help you get started. I’ve included a list in the slides above.
Let me know how it goes!
I’d like to conclude by saying that HTML has worked very well for us, and I strongly encourage you to give it a shot. Send me a message at chad@imvu.com to let me know if it works for you or not!
Q&A
I’d like to clarify some points I made during the Q&A session:
A man from Khronos approached me and said I was a bit hard on WebGL. If I came across that way, I didn’t mean it. I love WebGL and the work that Khronos has done, but I don’t think it’s prime time _yet_. We can’t port our existing product, which often runs on machines without a reasonable GPU, to WebGL yet. We do, however, intend to write a WebGL backend eventually. By the end of the year, we expect 50% WebGL adoption.
Someone asked about initial startup overhead. I said we probably consumed 30 MB of RAM and address space in base Gecko initialization, but that we’d done absolutely no work to reduce that. On the console, I expect it would be worth optimizing further.
There was some discussion around how our designers and engineers interacted. I mentioned that our teams are often cross-functional: our designers typically know enough CSS to make changes if they want and some of our engineers have strong design sense. We find it more productive to just sit down together and have an engineer and a designer work together over an hour to polish up the experience.
Thanks!
Thanks to Chuck Rector, Ben McGraw, Chris Dolezalek, Enno Rehling, Andy Friesen, James Birchler, Brett Durrett, Jon Watte, Laura Austin, Ben and Staci Scott, and Michael and Samantha Van Waardhuizen for providing valuable feedback during the preparation of these materials.
One thought to “HTML5 – New UI Library for Games [Annotated Slides]”