Web Analytics Maturity Model Q&A

This post is a translation/adaptation of 
Questions à Stéphane Hamel, concepteur du "Web Analytics Maturity Model" (WAMM), where I was interviewed by Nicolas Malo.
---
Stéphane Hamel is a leading voice for Web Analytics in North America and has recently published the Web Analytics Maturity Model (WAMM). The WAMM is the result of an 18 months study conducted as an MBA project at Université Laval in Québec-city. Stéphane is also a member of the board of the Web Analytics Association. As part of his European tour of the "Roadmap to online analytics success" he will be in Lille, France on March 18th and 19th (registration details for Lille are available on Hub'Sales site, other locations for the workshop are listed on the WAMM page).

Nicolas Malo: Bonjour Stéphane. Can you explain what is the WAMM for the uninitiated?

Stéphane Hamel : I love acronyms! Because I have extensive experience in e-business and measuring online performance, as an online tutor to UBC's Award of Achievement in Web Analytics and frequent conference speaker, Laval University suggested my final project for Master in Electronic Business Management (MBA ) be the creation of a full-semester, graduate level course on the topic of web analytics. I quickly realised such a course could not be a traditional approach of covering basic concepts or how to implement and use popular web analytics solutions. My practitioner and consulting experience with several clients as well as looking at the state of the industry also thought me there was obviously something wrong with the traditional approach since a majority of companies could not achieve the expected results. At the same time, throught years of MBA studies, it was clear companies were faced with many other difficult challenges. This led to a position that web analytics is primarily based on disciplines that have existed for a long time and that it can be easier.

Hence the essential questions: What are the success factors of web analytics? What are the critical process areas supporting such a success? How to determine if a company is on the right track to succeed, or rather at high risk of failing?

The WAMM proposes a maturity model based on Critical Success Factors leading to the effective use of analytics to make informed decisions and optimize online business processes.

Nicolas Malo: How did you come to the conclusion that it was necessary to formalize a model of maturity in Web Analytics?

Stéphane Hamel: Some of my colleagues in the industry doubt the merits of a maturity model on the basis that each company is so different that models simply do not work. A model is not a dogma, a model is not perfect ... for example, meteorological models can hardly predict the weekend weather! Yet, in most cases, they are very useful and continue to evolve to become more accurate. A model is a simplified representation of a more complex reality. What is included stems from hypothesis and observations of what we understand of a concept. It evolves in response to advancements and development of new theories, deeper understanding and validation in the field.

The element of "maturity" in the model describes the essential elements and relationship of various domains of expertise to gradually move from a rather chaotic and improvised approach to a mature, structured and disciplined one.

One of the major benefit of a maturity model is to spark discussion and reflection within the organization. It allows the team to put objectives against the current situation and act accordingly. Once convinced of the merits of measuring performance of its activities, the organization needs a road map to avoid the pitfalls that so many others before them have encountered.

Nicolas Malo: Is it just an academic exercise or you leveraged real life experience to define the different dimensions?
Stéphane Hamel: The academic side brought a structure and disciplined approach, but the WAMM is primarily a field study. When attending conferences, mostly in the United States, I was always a bit disappointed to hear great success stories then come back to reality and realize we were so archaic compared to the Amazon, eBay, Expedia, Dell and others flag ship.

I have the pleasure of working with several clients of various sizes and verticals in Quebec, Canada and abroad: automotive, insurance and finance, retail, travel, non-profit, etc. Each of them exhibit specific characteristics, yet they face very similar challenges when it comes to web analytics. The concept of maturity levels arose from a desire to understand the commonalities of those situations.

Key process areas, or dimensions, stems from the study of several models offered by various industry experts and vendors, other related fields such as business analysis and data warehousing, as well as academic concepts encountered throughout the MBA studies.

Nicolas Malo: If there was one strong idea to retain from WAMM, what would it be?

Stéphane Hamel: The first key process area, the one that will have the strongest influence on the success of your web analytics initiatives, is the management, governance and the degree of adoption of analytical practices in the organization. In second place comes the way objectives are defined and the scope of intervention. While it is interesting to debate about the tools and their multitudes of features, they actually have very little impact on success. Yet, such findings are not new and have been demonstrated numerous times in the past...

Nicolas Malo: You recently announced that you're writing a book out of the WAMM project. Can you tell us more?

Stéphane Hamel: The whole study project is quite big and I had to extract a summary in order to solicit feedback from the web analytics community. I invite you to take a look at immeria.net/wamm to download the paper and provide feedback. My original plan, to design a full-semester, graduate level course for Université Laval, is available since January under the name "MRK-6005 - Analytiques Web". There are many books on the topic of web analytics, but few (none?) academic book with a the typical structure of text books. Unfortunately, it will take a while to write, but the course structure and material is complete, the workshop is ready and has already taken place in Ottawa, and soon in Montreal (Feb 10th).

Nicolas Malo : Thank you Stéphane for this interview! A final word?

Stéphane Hamel: Web analytics is a relatively young and in this regard, it often reminds me of the heyday of the early Web. The pace of innovation is impressive, but the most remarkable aspect is probably the collaboration, mutual support and friendship that comes from our interest for web analytics. As I often say, "social media" is cool, but there is nothing like meeting people in person. Therefore I will be in Lille in March (as well as several other cities in Europe, Canada and the US)!

Note : I will actually be in Lille for the European tour of my workshop "Roadmap to online analytics success". I will also participate in a panel on the theme "making the right technological choices in web Analytics" at the Nord IT Days conference and one of the main speakers at the "Rendez-vous des Web Analytics" in Lille Thursday, March 18, 2010.

Of data and intuition; web analytics models and frameworks

We strive to convince managers of developing a data-driven culture instead of relying on what seems to be emotional, arbitrary and political decisions. We despise the HiPPO and try to tame the beast in various ways. Data, experience and some intuition have a role to play in making good business decisions. As Jim Sterne puts it in Data, Data Everywhere and Not a Thought to Think, "Data is good. Data is valuable. Just don’t be black and white about it. You can use data to help you and you can use data to trap yourself."

Training & education: again!

Developing critical thinking is one of the primary role of education - once you understand the fundamentals behind something, you can more easily extend your knowledge to more complex, difficult and unexpected situations. There is an ongoing debate between those who think quick hands on experience with specific tools is a better career strategy than getting educated about concepts of marketing, statistics, management and online strategies.

As with data, nothing is black or white.

Implying the only value businesses are looking for are tools masters would be a huge mistake. As web analytics evolves from marketing centricity to become online analytics and business analysis, would-be web analysts are faced with a dilemna: get hands on experience with specific tools to quickly get a job or gain more education to master the core concepts of doing business online.

Of course you need to get started in analytics and experience is key, but everyone knows knowledge is power. In my opinion, keeping less experienced professionals entrenched in a role of "tool X expert" is a short term strategy to get a job - not necessarily a bad one - but certainly not a long term career objective.

Data vs intuition

Jim Sterne's most recent post cautions us about the blind faith in data and the role of intuition in the business management decision process.

The New York Times touched on this topic some time ago:  “debate between intuition and empiricism is as old as Plato, who thought that knowledge came from intuitive reasoning, and Aristotle, who preferred observation. The argument has seemed especially intense lately, as one field after another has struggled to define the role of human judgment in a data-saturated society… These disagreements can sometimes be exaggerated, because everyone agrees that intuition and empiricism both have a role to play. But the fight over how to balance the two is a real one.”

Turns out this article talks about the field of medicine, not web analytics!

From ad hoc to structured

In the same New York Times article, a doctor is quoted as saying "You cannot write a protocol that perfectly fits any patient" - just like a model purpose isn't to be taken as the universal truth. Yet, doctors spend years studying factors leading to countless diseases and medical conditions. Maybe more than any other discipline, medicine is hugely dependent on "models" and "frameworks", or more commonly called "clinical protocols". Of course, not everyone can or wants to be a doctor.

Even if there are some critics against the value of maturity models, they are inevitable. Eventually our beloved industry will become more commoditized and replaceable because that is the nature of business evolution. Tools will continue to quickly evolve, but as Jim Novo brilliantly exposed in a recent forum post: people should learn to "think for themselves about analytical problem solving in a business environment, to develop their own creativity in problem solving rather than following robotic formulas that may lose relevance over time".

Models, frameworks and expert systems

Voltaire said “it is dangerous to be right in matters on which the established authorities are wrong”. Surprisingly enough - or maybe not - according to Wikipedia, the words "model" and "framework" share some similarities in their attempt to create a description of a complex entity or process.

Model: is a pattern, plan, representation, or description designed to show the main object or workings of an object, system, or concept.
Framework: a basic conceptual structure used to solve or address complex issues, usually a set of tools, materials or components.
Source: Wikipedia

Admittedly, both terms suffers from being dismissed as buzzwords and fashions.

Proposing a framework without a conceptual model is a way to keep a mystical aura around the underlying concepts and justifying claims that web analytics is hard. A framework without a model is like doing the grocery without a menu: you will eat something, but the results won't be optimal. You will not only waste valuable resources, it will also cost you more.

From model to framework, to decision support

If you have read the Web Analytics Maturity Model you should have received an email soliciting your feedback. The volume of constructive comments reinforce my opinion that WAMM is proving to be a valuable tool to asses the maturity of web analytics.

Pushing the concept from model to framework, we can envision the possibility of an expert system that would assist analysts - relieving them of trivial tasks and growing their expertise through knowledge sharing and collaboration. Akin Arikan, from Unica, suggested we should develop a Clinical Decision Support System for web analysts, just like those found in the field of medicine. I think this is a valuable objective.

Defining models, frameworks and expert systems requires a robust approach. Something more often found in academic and scientific research than spontaneously surfacing out of commercial interests. They need to be independent of too strong a vendor or consulting influence. Once initially defined, they need to be put to the test, be peer reviewed, become as broadly accepted as possible and evolve over time.

Next steps for the Web Analytics Maturity Model

Socrates said "There is only one good, knowledge, and one evil, ignorance". He also said "The only true wisdom is in knowing you know nothing".

• I love to get your feedback! Peer review is an essential element of the process.
  
• I'm also working on methods to gather empirical data while the model continues to be adopted by web analytics practitioners, consultants and agencies around the world.
• Workshops and conference sessions are being planned in several cities.
• And lastly, I have to keep the hood on one last element that will be available to those who wants to get more involved.

Workshop: Roadmap to online analytics success

Increasing conversions, reducing acquisition costs and optimizing online marketing activities are often cited as key reasons for venturing into web analytics. Properly installing the tool is a challenge in itself but even when done successfully, unexpected issues surface and the value of online analytics quickly fade.

Analysis and data has been part of my job ever since I started over 20 years ago. I spent most of the last 15 years as a practitioner or consultant developing web sites and strategies for dozens of companies. More recently, I spent 18 months studying why some companies succeed while so many fail at web analytics. The result was the release of a first proposal for a Web Analytics Maturity Model.

Far from pretending to be smarter or more "expert" than others, I guess it's a matter of interest, lots of grey hair and a certain level of freedom that allows me to spend the time documenting and sharing my experience and expertise.

Several people asked for more. The answer is a half-day or full-day workshop I will present in several cities, starting with:

• Ottawa, Jan. 20th (with PublicInsite)
• Montreal, Feb. 10th (+special event with Jim Sterne in the afternoon, with the help of CEIM and TechnoMontreal)
• Lille, France, March 17th or 18th* (Hub'Sales)
• Antwerp, Belgium, March 23rd* (TheseDays)
• Amsterdam, March 25th* (TheseDays)
• Toronto, April 6th (eMetrics)

I'm always open to work with local partners to bring this workshop to a city near you. Contact me if you are interested to attend or organize.
* Registration coming soon

Workshop description

This course proposes an efficient methodology that addresses online analytics from a managerial perspective. The goal isn’t to learn how to use a tool or optimize your latest marketing campaign from a tactical perspective. The methodology stems from years of experience in online strategies and analytics as a practitioner and consultant to several clients as well as an 18 month study of critical success factors of analytics.

You will learn about the six key process areas of a successful online initiative and how analytics can be leveraged at every step of the site life-cycle. You will get a clear road map to achieve success.

Objectives

This workshop will go through an assessment of your online analytics goals and objectives from a business, marketing and web executive's point of view. You will learn about the critical success factors and the process that will enable you to plan for future advancement. At the end of this workshop, you will have learned:

• A method to do a strategic evaluation of your current and desired situation.
• How to leverage the six critical process areas of a data driven organization.
• Tips & tricks to define realistic objectives aligned with your business... how to measure & achieve them!
• How to identify areas requesting more investment and energy.
• How to communicate effectively, be a change agent and overcome political storms.
• Several real-life examples, failures & successes.

Target audience

The course is specifically designed for analysts who are change agent in their organization or web and marketing managers who wants to leverage online analytics. For exampe, the workshop will be particularly appealing to:

• Web analysts and managers
• Marketing and online marketing managers
• E-commerce managers
• Directors
• IT directors and managers

Prerequisites

• No knowledge of web analytics or a web analytics solution is required
• No technical knowledge is required
• Interest for online strategies and measuring success is a must!

Workshop schedule

1. Introduction: analytical management = enlightened management
   
   • Overview of the online analytics maturity model
   • Description of each of the six critical process areas
   • Task: self-assessment of your current analytical situation
2. Defining objectives: ideas more than you can handle
   
   • Defining objectives: a process
   • Planning and opportunities prioritization
   • The role of user-centered design, personas and persuasion architecture
   • Task: business objectives vs online objectives
     
3. Statistics like you’ve never seen them before!
   
   • Average, trends, control limits from a management perspective
   • Population & segmentation
   • Correlation, trinity and multiplicity at work
   • A scientific approach to analysis
   • Conversion revisited
   • Task: Challenge!
     
4. Communication: tell me a number and I’ll tell you a story
   
   • Key Performance Indicators, what they are (and are not)
   • Dashboards has they should be
   • Task: what’s your KPI?
5. The analysis process
   
   • Problem solving and critical thinking
   • The importance of business analysis in analytics
   • SixSigma & Lean applied
6. Management
   
   • Managing the unmanageable: the organizational culture
   • Change management
   • Political factors
   • The ultimate team
   • Task: case study
7. Conclusion
   
   • Wrapping up
   • Innovation & the future of analytics
     

The workshop is filled with anecdotes, tips & tricks gleaned throughout over 20 years of experience, the last 15 spent working on dozens of online projects strategies and hands on web analytics learning.

Course material

Presentation hands out with annotations and exercises.

Additional info

Visit the Web Analytics Maturity Model section on immeria.net.

About the instructor

Stéphane Hamel is a leading voice for online analytics, helping businesses understand the value of performance measurement and process optimization. With over twenty years of experience, most spent developing web sites and online strategies, he is now teaching a full-semester, graduate level class about online analytics at Laval University (Québec City), as taught over 500 people through lecturing the « Award of Achievement in Web Analytics » and « Introduction to Business Analysis » classes at the University of British Columbia. He is a member of the « International Institute of Business Analysis » and on the board of directors and treasurer of the « Web Analytics Association ». Stéphane received the « Web Analytics Association Leadership and Technical Excellence Recognition » and is a frequent speaker at the « eMetrics Marketing Optimization Summit » and other conferences. Stéphane owns a Master in Business Administration specializing in eBusiness.

Nutty's creativity: 2nd grader analytics

Here's the answer to a little question I asked on Twitter. It's straight from a math textbook for 2nd graders. It's a nice little demonstration of critical thinking and how sometimes, the obvious answer might not be the best one.

Nutty's creativity

M.Mansa, professor in a 2nd grade elementary class, asked: "Which number doesn't fit in the following series? 2, 4, 5, 6, 8"

Every student found the right answer, except one, Nutty. It's 5, everybody knows that!
Except Nutty, who says "I thought about it, but I prefer 4 because..."
Unhappy, Prof. Mansa assign Nutty to write a text on "Why I should have chosen 5". Here's what he wrote.

"At first, I thought about 5 because it's the only odd number. But why not 8? It's the only one drawn from one closed loop line. For me, the best answer to a problem is often the 2nd one. It shows I can go beyond the obvious and be creative.
After, I thought about 2, by replacing it with 7, we would end up with a nice suite of 4,5,6,7,8...
I looked at 6, but didn't see anything special about it. Then I thought... Euréka! It's the only one in the series than can be reversed to create another number.
At last, I considered 4. 4 is the only one drawn from straight lines, without any curves. It's also my preferred number because April 4th (4th of the 4th month) is my birthday.
I think I should have chosen 5... as everyone else. After all, it's the only one that can't be added to another number of the series to sum up to 10: 2+8=10, 4+6=10...
Ok, I'll take 5!"

Reference: "Défi mathématique, 1er cycle, #2", Cheneliere/McGraw-Hill, Michel & Robert Lyons.

Relation to analytics

Wikipedia defines analytics has "the science of analysis, or how an entity (i.e., business) arrives at an optimal or realistic decision based on existing data." The series above is a nice little data set. Drilling down on the meaning of analysis, we see it's "the process of breaking a complex topic or substance into smaller parts to gain a better understanding of it." The breakdown of the above question is a nice example. We've seen how data can be used to tell carefully crafted stories that are absolutely valid given the right context. It relates to one of the seven online analytics axioms defined by my friend Christopher Berry: "It is possible for there to be two optimal, equally true, answers to a problem."

Some people have harder times developing critical thinking. Despite the fact employers often look for experienced, hands-on professionals, the path to become more than a number cruncher and report junkie is to develop expertise in all of the key process areas of analytics. Critical thinking being one of the essential skills of a good analyst.

My advice is this: in 2010, gain more hands on experience, but also strive to develop your understanding and mastering of the underlying concepts of online analytics.

Happy 2010!