IMVU Memcached Usage at 150k Requests per Second per Node

IMVU Engineering 3 Comments

By: Jon W

Running a large web site is quite interesting. Not only do you have a *lot* of customers to keep happy, but you also have a *lot* of hardware to keep happy. Racks and racks of servers go into making sure that IMVU is always available, 24 hours a day, 7 days a week, except for occasional brief outages.

One of the kinds of servers we use is known as “memcached,” which is a simple way of reducing the workload for database servers by caching the result of frequent database queries in memory. This has two benefits:

  • It reduces load on the database servers (which are expensive) and moves it to machines that just have a lot of memory (which is cheap).
  • It allows us to serve web pages faster. Each web page within imvu.com is the result of several database queries; if the result of all of those queries is already in memory somewhere, we can send the page back in less than a second, whereas it would take longer than that if we had to go to the database for each query.

That being said, it’s no surprise that we have a significant number of servers running memcached to improve the speed and performance of our site. It used to be that when your memcaches hit 50,000 requests per second, they would top out — but that was a couple of years ago. It also used to be that you would switch to (potentially lossy) UDP for memcached traffic, rather than staying on (potentially laggy) TCP. However, for better or for worse, we’ve stayed on TCP for our memcached traffic for now.

It’s Hack Week at IMVU

IMVU Engineering Leave a comment

By: James Birchler and Roland Blanton

Yesterday we kicked off another Hack Week at IMVU, a solid week when we put product development in the hands of IMVU engineers. What does this mean? An engineer can spend the week working on something they personally feel is valuable to the company. It’s a way to harness experience and insights from across the company and give everyone more ownership over what we are building here. The buzz in the building is tangible: there are fewer meetings, less process around group work, and people are focused on finishing their features to put them in front of customers.

Hack Week has been an integral part of our engineering culture since 2007, giving our software engineers a chance to guide product development and test their ideas. This tradition has resulted in many popular features like Outfits Management, Turbo Product Loading of 3D assets, IMVU Badges, and shopping directly from a 3D chat. All these features were driven by IMVU engineers during past Hack Weeks and then adopted by our product teams for release to all customers.

To help foster an environment of creativity, we use our A/B experiment system to make it easy and low-risk for us to test product innovations with customers. Rather than rely on the opinions in the room, we prefer getting feedback directly from customers to help guide our decisions.

In order to maximize chances for success, we follow some lightweight processes and rules:

  1. The goal in most cases is to deliver valuable features live to customers in experiments by the end of the week.
  2. Engineers choose projects to work on–sometimes from a team’s existing product backlog, and sometimes not.
  3. We work closely with product owners, user experience designers, technical leads, QA engineers, and other stakeholders to come up with what we think is a good plan.
  4. We start hacking, ultimately releasing features in A/B experiments to our customers.
  5. We only work on one project at a time (it’s pretty easy to find yourself starting many projects and never finishing, which runs counter to our overall goal of delivering value to customers).
  6. Everyone does a demo of their work at the end of Hack Week.

There is a lot of face to face, ad-hoc collaboration going on in the weeks preceding Hack Week, and during Hack Week itself.  The week concludes with demos to the entire company, a strong feeling of engagement with our customers and our product, and curiosity about what our customers will tell us about what we’ve built.