Practical analytics: incoming traffic quality

The initial inquiry was something like this: "In Site Catalyst how do i calculate bounce rate for referrers? I am trying to figure out the quality of referrers to the site using bounce rate by referrers".

Step 1: Define

The first analysis step is to define the improvement goal that is consistent with customer demand and business strategy.

We can rephrase that into something closer to a SMART objective: "During the month of October, which referrers brought people who engaged with our site beyond the initial page?". This might be good enough, but the initial statement was about "quality", which ultimately needs to relate back to the business primary objective (or one of the secondary ones). For an ecommmerce site, it could be: "During the month of October, which referrers brought customers?". We could even qualify it further by stating a minimal purchase amount, life time customer value, etc.

Step 2: Measure

Here we have a very good example of the difference between reporting and analysis. Reporting would limit itself to each individual metrics such as Referrers and Visits and leave it to the reader's imagination to find a meaning to those numbers. Analysis aims to look at the correlation between various metrics and build up a good story around them.

In the second step, we want to look at the process and collect relevant data for comparison.

In this case, we have a couple of important elements we can look at:

• Visit: "A visit is an interaction, by an individual, with a website consisting of one or more
  requests for an analyst-definable unit of content (i.e. “page view”)."
• Page Views: "The number of times a page (an analyst-definable unit of content) was viewed."
  
• Referrer: "The referrer is the page URL that originally generated the request for the current page view or object." In our case, we will want to look at the originating domain, not the whole URL. So we will talk about the "referring domain" for "external referrers".
  
• Conversion: "A visitor completing a target action." Typically, it's easier to look at the referring domain that brings us the most conversions.
• Entry Page: "The first page of a visit."
• Page Views per Visit: "The number of page views in a reporting period divided by number of visits in the same reporting period."
  

Step 3: Analyze

We want to analyze the relationship and causality of various factors.

In order to analyze what is happening, we need to define two new metrics derived from the basic metrics commonly available in any web analytics solution:

• Bounce Rate: The number of visits that resulted in a single page view and then left the site. It can be defined in Site Catalyst as a new calculated metric of Single Access/Entries.
  
• Weighted Bounce Rate: Same as above, but gives more weight to pages that are viewed more often, thus pushing problematic pages to the top of the list. It is calculated from this formula: (Single Access/Entries) * (Page Views/Total Page Views).

In Site Catalyst, the only way to find out Bounce Rate by Referrers is to go under Paths/Pages/Most Popular Pages and select our newly defined Bounce Rate and Weighted Bounce Rate.

Then we can simply click on the correlate icon and select Finding Methods/Referring Domains.

Step 4: Improve

Optimize based upon the analysis and various design experiments.

Now we have all information at hands to make hypothesis and validate them. One could think we don't have much control over who links to our site. In this case, we found out some referrers were simply linking to a "deep" page while they should link to a page that comes up sooner in the process. Since those referrers were mostly partner sites, we could simply ask them to fix the links. They are actually sending us qualified traffic, but at the wrong step of an important conversion process! End result: frustrated customers, loss of revenue.

Step 5: Control

The last step is to control that our changes results in positive outcomes from a user perspective, but also from a business point of view. We can simply run A/B reports showing the months of October and November side by side.

We want to ensure that any variances can be explained. Set up audit at specific intervals to asses conformity and institute control/correction mechanisms.

Six Sigma

Here it is! Without knowing it, we've just went through a very simplified Six Sigma approach that can easily be applied to web analytics. Using the Define-Measure-Analyze-Improve-Control process, it is easier to keep the business objectives in mind and stay focused on actionable data that brings tangible outcomes.