Friday, January 22, 2010

Of data and intuition; web analytics models and frameworks

We strive to convince managers of developing a data-driven culture instead of relying on what seems to be emotional, arbitrary and political decisions. We despise the HiPPO and try to tame the beast in various ways. Data, experience and some intuition have a role to play in making good business decisions. As Jim Sterne puts it in Data, Data Everywhere and Not a Thought to Think, "Data is good. Data is valuable. Just don’t be black and white about it. You can use data to help you and you can use data to trap yourself."

Training & education: again!

Developing critical thinking is one of the primary role of education - once you understand the fundamentals behind something, you can more easily extend your knowledge to more complex, difficult and unexpected situations. There is an ongoing debate between those who think quick hands on experience with specific tools is a better career strategy than getting educated about concepts of marketing, statistics, management and online strategies.

As with data, nothing is black or white.

Implying the only value businesses are looking for are tools masters would be a huge mistake. As web analytics evolves from marketing centricity to become online analytics and business analysis, would-be web analysts are faced with a dilemna: get hands on experience with specific tools to quickly get a job or gain more education to master the core concepts of doing business online.

Of course you need to get started in analytics and experience is key, but everyone knows knowledge is power. In my opinion, keeping less experienced professionals entrenched in a role of "tool X expert" is a short term strategy to get a job - not necessarily a bad one - but certainly not a long term career objective.

Data vs intuition

Jim Sterne's most recent post cautions us about the blind faith in data and the role of intuition in the business management decision process.

The New York Times touched on this topic some time ago:  “debate between intuition and empiricism is as old as Plato, who thought that knowledge came from intuitive reasoning, and Aristotle, who preferred observation. The argument has seemed especially intense lately, as one field after another has struggled to define the role of human judgment in a data-saturated society… These disagreements can sometimes be exaggerated, because everyone agrees that intuition and empiricism both have a role to play. But the fight over how to balance the two is a real one.”

Turns out this article talks about the field of medicine, not web analytics!

From ad hoc to structured

In the same New York Times article, a doctor is quoted as saying "You cannot write a protocol that perfectly fits any patient" - just like a model purpose isn't to be taken as the universal truth. Yet, doctors spend years studying factors leading to countless diseases and medical conditions. Maybe more than any other discipline, medicine is hugely dependent on "models" and "frameworks", or more commonly called "clinical protocols". Of course, not everyone can or wants to be a doctor.

Even if there are some critics against the value of maturity models, they are inevitable. Eventually our beloved industry will become more commoditized and replaceable because that is the nature of business evolution. Tools will continue to quickly evolve, but as Jim Novo brilliantly exposed in a recent forum post: people should learn to "think for themselves about analytical problem solving in a business environment, to develop their own creativity in problem solving rather than following robotic formulas that may lose relevance over time".

Models, frameworks and expert systems

Voltaire said “it is dangerous to be right in matters on which the established authorities are wrong”. Surprisingly enough - or maybe not - according to Wikipedia, the words "model" and "framework" share some similarities in their attempt to create a description of a complex entity or process.
Model: is a pattern, plan, representation, or description designed to show the main object or workings of an object, system, or concept.
Framework: a basic conceptual structure used to solve or address complex issues, usually a set of tools, materials or components.
Source: Wikipedia
Admittedly, both terms suffers from being dismissed as buzzwords and fashions.

Proposing a framework without a conceptual model is a way to keep a mystical aura around the underlying concepts and justifying claims that web analytics is hard. A framework without a model is like doing the grocery without a menu: you will eat something, but the results won't be optimal. You will not only waste valuable resources, it will also cost you more.

From model to framework, to decision support

If you have read the Web Analytics Maturity Model you should have received an email soliciting your feedback. The volume of constructive comments reinforce my opinion that WAMM is proving to be a valuable tool to asses the maturity of web analytics.

Pushing the concept from model to framework, we can envision the possibility of an expert system that would assist analysts - relieving them of trivial tasks and growing their expertise through knowledge sharing and collaboration. Akin Arikan, from Unica, suggested we should develop a Clinical Decision Support System for web analysts, just like those found in the field of medicine. I think this is a valuable objective.

Defining models, frameworks and expert systems requires a robust approach. Something more often found in academic and scientific research than spontaneously surfacing out of commercial interests. They need to be independent of too strong a vendor or consulting influence. Once initially defined, they need to be put to the test, be peer reviewed, become as broadly accepted as possible and evolve over time.

Next steps for the Web Analytics Maturity Model

Socrates said "There is only one good, knowledge, and one evil, ignorance". He also said "The only true wisdom is in knowing you know nothing".
  • I love to get your feedback! Peer review is an essential element of the process.
  • I'm also working on methods to gather empirical data while the model continues to be adopted by web analytics practitioners, consultants and agencies around the world.
  • Workshops and conference sessions are being planned in several cities.
  • And lastly, I have to keep the hood on one last element that will be available to those who wants to get more involved.