In this example, I will use Omniture SiteCatalyst but the same type of analysis could be done with any solution. For the sake of this demonstration, let's assume it's a B2B site, so visitors comes from very specific partner companies.
I was faced with a very good case where analysis, instead of reporting, made all the difference. Take a simple statement: 15% of the visitors to the web site use Windows 2000/IE 5. We know this could lead to slightly different design considerations.If we are simply reporting, we're stuck with 15%... is it worth designing for this platform? How can we tell?
Since we suspect Windows 2000/IE 5 use might be declining, let's look at the usage trend for our visitors. We already have a clearer indication, going from nearly 16% down to 12% in 5 months. Plus the fact Omniture can tell us the Internet average is about 6%, indicating our site receives a larger than average share of Windows 2000 visitors.Could there be just a handful of visitors (companies) using Windows 2000 and influencing this reporting? Now we need to create a Custom Datawarehouse Report using Omniture's Advanced Analysis. We can ask for the number of visits by OS, grouped by Company, with a breakdown by month. Once we get the CSV file loaded in Excel, a simple Pivot Table can give us a very good look at what is going on.
Now we can spot that only a couple of companies, some of them representing a fair share of the traffic, have much higher usage of Windows 2000 (42% in the above example). This makes sense because it appear one of those companies is a banking environment, and we know those corporate environment traditionally takes longer to adopt more recent technologies. Furthermore, we see there's also a clear negative trend, from 122 visits in October to 54 in February (while XP is generally growing).
Now that we have done a more thorough analysis, we're in a much better position to decide if we need to invest in designing for this platform. From a simple report stating "15%" with very limited context, we were able to identify more detailed behavior and address the impact on a limited subset of our visitors.
In the spirit of the 10/90 rule coined by Avinash Kaushik, getting the number was easy. The analysis is where the value is: how is the site structured? who's your audience? which other reports and metrics will help you get the right decision? Too often, people are complaining about the tool: it's not working, it's not configured correctly, it doesn't give me my numbers... I guess that's where experience and an analytical mindset can make all the difference.
(Ultimately, the recommendation was to avoid integrating costly exceptions in the design and the development process.)