Tuesday, September 1, 2009

Review of Maturity Models

This is a long post part of the Web Analytics Maturity Model (WAMM) series. This time, I'm looking into six maturity models from various domains:
  • WebTrends Digital Marketing Matutiry Model (DM3) (Web analytics vendor)
  • Gartner’s Maturity Model for Web Analytics (Market analyst/Information technology)
  • Capability Maturity Model Integration (CMMI) from the Software Engineering Institute at Carnegie-Mellon University (Academic/Software engineering)
  • The Data Warehousing Institute Business Intelligence Maturity Model (Association/Business intelligence)
  • Stratigent Marketing Analytics Model (Web analytics consulting)
  • Competing on Analytics maturity by stage, by Thomas Davenport in "Competing on Analytics: the new science of winning" (Academic)
Those models will serve as inspiration to develop the Web Analytics Maturity Model.

Previous posts in the WAMM series:
  1. Overview of the Web Analytics Maturity Model 
  2. Definition of Web Analytics 
  3. Components of the Web Analytics Maturity Model
For more information visit the Web Analytics Maturity Model area on immeria.net

WebTrends Digital Marketing Maturity Model (DM3)

The DM3 model, announced in the spring of 2009, claims to offer a "framework and objective criteria to determine the sophistication of an organization’s measurement and analysis skills, staffing and best practices across multiple digital marketing disciplines, from web sites to social media". The proposed model defines six core areas, each one graded on a four level scale.

In their model, WebTrends consider the first "pillar" to be the most important one, and most pillars are further sub-divided.
  1. Measurement strategy: A measurement strategy is in place and aligned with business objectives.
  2. Analytics resources and domain expertise: The number of employees dedicated to measurement and analysis, resources allocation and problem resolution skills.
  3. Data integration and visualization: Establish correlations between multiple data sources and communication of insight through the organization, presentation and delivery of data.
  4. Data analysis and insight: Skills to turn web-based data into insight.
  5. Adoption and governance: Role-based training, change management, security, data consistency and quality and process-driven governance.
  6. Ongoing optimization: A continuous improvement process to identify opportunities and test various optimization scenarios is in place.
Still in beta, this model certainly needs further refinement. The sub-division and ratings are based on strong experience but lack empirical approach and supportive evidence. The desire to push for an industry standard is certainly good but the actual approach centers heavily around this vendor consulting services. As others have criticized, the spider-chart is "making an assumption that certain groups of activities come together" and "gives the impression that adjacent pillars are somehow correlated because the area that is formed by joining their scores depends on both scores and doesn’t give each one visual independence of the other".

More info: WebTends Digital Marketing Maturity (DM3)
Image credit: WebTrends

Gartner Maturity Model for Web Analytics

This model was first published in 2008 and is aimed at "CIOs and other IT and business leaders responsible for the strategic success of their organization's Web optimization strategy". In this model, Gartner analyst Bill Gassman reiterates the fact "the problem tends to be a lack of skills and processes rather than issues with the tools themselves".

This model focuses on the responsibility to pick the right solution (or set of solutions) for the stated business goals. It is acknowledged that organizations can demonstrate a level of maturity for some tasks while not necessarily mastering everything at a specific level. These levels are merely guidelines to set goals and analyze the gap between current and desired state. Although being very pragmatic and straightforward, it addresses only one dimension of a successful analytical organization: the web analytics solution technologies and their usage sophistication.

In its DM3 model, reviewed later, WebTrends criticize the Gartner model as failing to "address social media, SEM and other digital marketing measurement channels". However, looking beyond the specifics of each level, managers should understand the concepts and level of sophistication demonstrated at each level, and therefore, easily acknowledge social media, SEM, mobile and such are covered at "Level 4 - Collaborative" of the Gartner model.

More info: Gartner Maturity Model for Web Analytics
Image credit: Gartner


Capability Maturity Model Integration (CMMI)

CMMI was originally developed in 1989 as a tool for objectively assessing the ability of contractors’ processes to perform a contracted software project.

The main point of CMMI is the objective evaluation of the “ability to perform” and as been applied to many areas beyond technology and engineering, notably risk management and business process optimization.
  1. Ad hoc (chaotic): Typically undocumented and in a state of dynamic change, tending to be driven in an ad hoc, uncontrolled and reactive manner by users or events.
  2. Repeatable: Some processes are repeatable, possibly with consistent results. Process discipline is unlikely to be rigorous, but where it exists it may help to ensure that existing processes are maintained during times of stress.
  3. Defined: Sets of defined and documented standard processes established and subject to some degree of improvement over time. These (as-is) standard processes are in place and used to establish consistency of process performance.
  4. Managed: Using process metrics, management can effectively control the actual process. In particular, management can identify ways to adjust and adapt the process to particular projects without measurable losses of quality or deviations from specifications.
  5. Optimizing: Focus is on continually improving process performance through both incremental and innovative technological changes/improvements.
CMMI has been criticized as being "overly bureaucratic and promoting process over substance" and being a "classical engineering approach that does not take under consideration numerous human cognitive, organizational, and cultural factors essential for the success of every project". Those criticisms should serve as a reminder in defining a simple, realistic and readily applicable model for web analytics.

More info: Capability Maturity Model Integration (CMMI)


TDWI Business Intelligence Maturity Model

Wayne Eckerson is the creator of The DataWarehouse Institute (TDWI) BI Maturity assessment tool, Director of Research and author of "Performance Dashboards: Measuring, Monitoring and Managing Your Business". The TDWI model main purpose is to "gauge where your datawarehousing initiative is now and where it should go next". Each of the six stages are defined by a number of characteristics: scope, analytic structure, executive perceptions, types of analytics, stewardship, funding, technology platform, as well as change management and administration.

The TWDI model addresses the business intelligence maturity, a term coined by Gartner analyst Dresner in 1989 as the "set of concepts and methods to improve business decision making by using fact-based support systems".
  1. Prenatal – Management Reporting: Standard set of generic reports distributed without discrimination for actual needs. Inflexible and hard to modify, users tend to bypass the established solution.
  2. Infant – Spreadmarts: Individual, disconnected spreadsheets and desktop solutions with their own set of data, metrics and rules that are not aligned with the organization. Although they offer a low cost and locally controlled solution, they prevent management from getting a clear and consistent picture of the organization.
  3. Child – Data Marts: All knowledge workers are empowered with timely information and insight. Data is shared and standardized at the department level, offering a standard set of application, business data and metrics.
  4. Teenager – Data Warehouse: Definitions, rules and dimensions are standardized across the organization and deeper analysis is available through interactive reporting and analysis. Queries crosses functional boundaries and the datawarehouse becomes a tactical tool to improve process efficiency across the whole value chain, contributing to a fact-based decision making culture.
  5. Adult – Enterprise Data Warehouse: There is now a single version of the truth, data becomes an asset as important as people, equipment and cash. Scorecards and dashboards contribute to align every worker with the corporate strategy. ROI becomes positive and new, unexpected ways of using this knowledge emerge as a competitive asset.
  6. Sage – BI Services: Data is open to customers and suppliers, extending the value chain beyond the corporate boundaries. Knowledge workers don’t have to switch context to analyze data since the data, information and insight is embedded into operational applications and contribute to decision engines (think of fraud detection, behavioural targeting, and automated applications). Business Intelligence becomes ubiquitous and value increase exponentially.
According to Thomas Davenport, analytics represent a subset of business intelligence. While BI can answer questions such as "what happened; how many, how often, where; where exactly is the problem; what actions are needed", analytics can answer "why is this happening; what if these trends continue; what will happen next; what is the best that can happen". From this perspective, we can asses a Web Analytics Maturity Model could borrow from the TDWI model.

More info: The Data Warehousing Institute Business Intelligence Maturity Model
Image credit: The Datawarehouse Institute

Stratigent Marketing Analytics Model

The Stratigent model outlines the four phases through which companies typically progress on the path to becoming data-driven organizations. A study conducted in 2007 revealed that:
  • A clear evolution of web analytics practices exists, and the progress of an organization can be ranked into one of several categories.
  • Internal education and adoption of analytic techniques are leading indicators of how quickly companies build competitive advantages in analytics.
  • Large budgets and big teams do not always correlate to success.

  1. Foundation Building: Focus on implementing web analytics. Limited executive support and no significant ROI. The availability of data spurs interest and builds momentum across the organization.
  2. Customization: The technology is customized to meet business-specific requirements. More relevant data spur adoption, drive ROI and secure funding for future investments.
  3. Optimization: Multivariate testing is introduced to gain additional insight about customer preferences and improve marketing program performance. Online data integration and the emergence of a continuous improvement process provide additional insight and set the stage for the next phase.
  4. Predictive Modeling & Targeting: Online and offline data are integrated to offer a complete view of the customer, allowing to deliver a highly targeted message for customers
The model is very interesting and highlights several aspects relevant to the definition of the Web Analytics Maturity Model:
  • A model can provide the vision, framework, and tactics necessary to put in place a web analytics program and develop a competitive advantage
  • Shows how web analytics can benefit marketing departments and contribute to a culture of analysis and insight
  • Provides a framework to demonstrate how the technology contributes to the organization’s capability to compete with analytics.
More info: Stratigent Marketing Analytics Model
Image credit: Stratigent

Davenport’s Maturity of Analytical Capability by Stage

Davenport, in "Competing on Analytics", proposes a maturity model encompassing three vital areas of a successful analytical company: organization, human and technology capabilities (what others refer as "people, process and technology") defined in 5 stages:
  1. Analytically impaired
  2. Localized analytics
  3. Analytical aspirations
  4. Analytical companies
  5. Analytical competitor
From a business perspective, "Competing on analytics" is without a doubt one of the best primer on the topic of analytics and its concepts can readily be applied to web analytics. Furthermore, some of the mentioned analytic competitors are well known for their online activities: Amazon, Google, Netflix, and Yahoo!, among others. For those reasons, the proposed model appears to be very close to the desired specification of the Web Analytics Maturity Model.

More info: "Competing on Analytics: The New Science of Winning"

For further information regarding the WAMM and its future evolution, including speaking, consulting and training, visit the Web Analytics Maturity Model area on immeria.net.

Coming up next: The Web Analytics Maturity Model!