Agile Business Intelligence: A Core Component of Agile Engineering

By Curtis Pullen

Business intelligence (BI) is not a term that is frequently brought up in the context of agile development – it seems more appropriate in the context of big, slowly moving companies crunching data on million dollar mainframes. The truth, however, is that business intelligence is a critical and integral part of the agile product development process. The goal of agile is to enable teams to react to feedback, and the goal of business intelligence is to provide that feedback.

Business intelligence in the traditional sense is unfortunately not equipped to provide the kind of feedback that an agile engineering team needs. When multiple teams of engineers are iterating in two or three week sprints, continuously deploying their changes to a production system as we do at IMVU, BI needs to be even faster. Those engineers are inventing new dimensions of potential analysis at an explosive rate and transforming user behaviour so quickly that data more than a few months old is of only archaeological interest, and to maintain that pace they need continuous feedback. They need to know when a mistake has been made so they can fix it; they need to know when they’re on to something revolutionary so they can run with it. They need BI that’s as agile as they are.

So what does agile BI look like? It employs many of the same patterns as agile software development: quick iteration, frequent releases, and close communication. While a feature is under development, BI analysts meet with product management to identify the high-level metrics that will best indicate the status of the project post-release, and with engineers, to determine how that data might best be obtained. When a project ships, BI analysts aggregate the data, translate it into meaningful information, present it, and seek feedback. Any questions or concerns raised by engineering or management trigger a new round of the cycle:

1. Discuss requirements with stakeholders.

2. Collect and interpret data

3. Deliver results to stakeholders and collect feedback

In order for this to succeed, BI needs to be tightly linked to both engineering and management. At IMVU, we accomplish this by having our engineers take on most of our BI work as an integral part of product development, while more sophisticated analysis is undertaken by a BI team with engineering support. We collect the right data exactly when we need it, we don’t waste time scrutinizing obvious patterns, and we illuminate the details as soon as it’s necessary.

Life moves fast, business intelligence should too.

This is an expanded version of some thoughts I put down in a contest for how best to describe agile BI. You can read the entries and then vote for mine here: http://www.pentaho.com/what_is_agile