All posts by imvuengineering

Agile Business Intelligence: A Core Component of Agile Engineering

By Curtis Pullen

Business intelligence (BI) is not a term that is frequently brought up in the context of agile development – it seems more appropriate in the context of big, slowly moving companies crunching data on million dollar mainframes. The truth, however, is that business intelligence is a critical and integral part of the agile product development process. The goal of agile is to enable teams to react to feedback, and the goal of business intelligence is to provide that feedback.

Business intelligence in the traditional sense is unfortunately not equipped to provide the kind of feedback that an agile engineering team needs. When multiple teams of engineers are iterating in two or three week sprints, continuously deploying their changes to a production system as we do at IMVU, BI needs to be even faster. Those engineers are inventing new dimensions of potential analysis at an explosive rate and transforming user behaviour so quickly that data more than a few months old is of only archaeological interest, and to maintain that pace they need continuous feedback. They need to know when a mistake has been made so they can fix it; they need to know when they’re on to something revolutionary so they can run with it. They need BI that’s as agile as they are.

So what does agile BI look like? It employs many of the same patterns as agile software development: quick iteration, frequent releases, and close communication. While a feature is under development, BI analysts meet with product management to identify the high-level metrics that will best indicate the status of the project post-release, and with engineers, to determine how that data might best be obtained. When a project ships, BI analysts aggregate the data, translate it into meaningful information, present it, and seek feedback. Any questions or concerns raised by engineering or management trigger a new round of the cycle:

1. Discuss requirements with stakeholders.

2. Collect and interpret data

3. Deliver results to stakeholders and collect feedback

In order for this to succeed, BI needs to be tightly linked to both engineering and management. At IMVU, we accomplish this by having our engineers take on most of our BI work as an integral part of product development, while more sophisticated analysis is undertaken by a BI team with engineering support. We collect the right data exactly when we need it, we don’t waste time scrutinizing obvious patterns, and we illuminate the details as soon as it’s necessary.

Life moves fast, business intelligence should too.

This is an expanded version of some thoughts I put down in a contest for how best to describe agile BI. You can read the entries and then vote for mine here: http://www.pentaho.com/what_is_agile

IMVU’s Startup Lessons Learned Conference Presentation

By Brett Durrett

IMVU presented at the Startup Lessons Learned Conference in San Francisco on April 23rd, 2010.  The event highlighted several companies that are being built using the “Lean Startup” framework created by Eric Ries, IMVU’s former CTO, largely based on his experiences at IMVU.

The conference was great and I had the opportunity to meet many smart entrepreneurs trying to build businesses out of great ideas.  I heard many stories about the challenges early startups encounter and could remember when IMVU was in that stage. I also talked to some people from companies that are now considered “big and successful” and heard a few comments along the lines of “been there, barely survived that”.

At the conference I had the realization that IMVU as a business is not exactly a “startup” anymore.   The goal of an early startup is discovering the right product and achieving a sustainable business model.  IMVU has been successful at this and is now all about building a growing, enduring business that is a high value for our customers and employees.  Though we still feel like a startup in many ways and hold onto the lean principles that proved to be so valuable, we now have new challenges to address that are typically not considered startup challenges.

A successful startup grows into a bigger business.  At IMVU, we heavily invest in our company so we can get more people working on features that delight our customers and build up the business.   With more people many of the ways you used to work don’t work anymore. For example, frequent meetings to get feedback from everybody in the company can work when you have fewer than 15 people… when you get to 50+ people this becomes a very expensive meeting. The overhead of making sure everyone in the meeting has background data and context to make an informed decision simply does not scale.  Joel Spolsky explained this well and provides good examples in his article, “A Little Less Conversation”.

There are a whole range of challenges in these transitions, from process to culture and all have to be accommodated as a company grows. At the conference I was approached by several people that had gone through the same experience, some successfully, some not.  I hope that some of what IMVU shared will help others to learn from our experience and allow more people to fall into the successful category.

Check out IMVU’s video presentation available at http://bit.ly/bBpUcm

If you just want to review the slides without audio, they are here: http://bit.ly/aGXqcY

IMVU Memcached Usage at 150k Requests per Second per Node

By: Jon W

Running a large web site is quite interesting. Not only do you have a *lot* of customers to keep happy, but you also have a *lot* of hardware to keep happy. Racks and racks of servers go into making sure that IMVU is always available, 24 hours a day, 7 days a week, except for occasional brief outages.

One of the kinds of servers we use is known as “memcached,” which is a simple way of reducing the workload for database servers by caching the result of frequent database queries in memory. This has two benefits:

  • It reduces load on the database servers (which are expensive) and moves it to machines that just have a lot of memory (which is cheap).
  • It allows us to serve web pages faster. Each web page within imvu.com is the result of several database queries; if the result of all of those queries is already in memory somewhere, we can send the page back in less than a second, whereas it would take longer than that if we had to go to the database for each query.

That being said, it’s no surprise that we have a significant number of servers running memcached to improve the speed and performance of our site. It used to be that when your memcaches hit 50,000 requests per second, they would top out — but that was a couple of years ago. It also used to be that you would switch to (potentially lossy) UDP for memcached traffic, rather than staying on (potentially laggy) TCP. However, for better or for worse, we’ve stayed on TCP for our memcached traffic for now.