It’s pretty much impossible to give a fair accounting of someone else’s methodology. You’re almost bound to simplify in the wrong places, give the wrong emphasis, miss the finer points.
This isn’t just a question of intention – unless you’ve really practiced a method your understanding is going to be intellectual not concrete – “thin” and not “thick.”
So instead of trying to compare Functionalism to someone else’s methods, I’m going to deal with our own experiences with what I take to be the most viable alternative direction – behavioral visitor segmentation – and explain why, important as I think it is, it isn’t the primary focus of our methodology.
I’ve already mentioned in Part I of this series that our background is in advanced visitor segmentation, and I’m going to base the essentials of my discussion on some work we did in the last couple years for an online travel site.
We were looking at personalization opportunities for this site. Since the client could give us rich event data in analyzable format, we used the Neural Segmentation tools that we’d developed some years before. To develop the visitor segments, we created a range of highly-specific variables related to usage of the site. These included the “generic” metrics like visits, time and views. But these were negligible in their effect. More important were metrics about types of tools used, searches, travel types, trip durations, hotel classes, discount levels, number of travelers, etc. Then there were a class of specially derived variables that captured unique behavioral cues, things like: destination flexibility, date flexibility, and “gaming” behaviors. All of these behaviors were evaluated using a neural technique called a self-organizing map. In the process, we produced a 6×6 grid (one of the largest we’ve ever decided on – more typical is 5×5) or 36 distinct visitor segments each with a distinct pattern of behaviors. This work formed the bedrock of our personalization and analytic suggestions.
This all sounds pretty good, doesn’t it? So why don’t I think this is a good technique? I do. What I don’t think, is that it’s a good general formal methodology.
Here’s why:
- Visitor segmentation is completely unique to every site – the variables, the behavioral cues and the type of segmentation is non-standard and we have no good rules (other than practitioner instinct) that really guide the process.
- Many sites don’t afford clear behavioral cues like this – so that the visitor segmentation is mainly concerned with the “Generic” usage variables. We have had very negative experiences trying to do analysis with usage segments.
- The tools to do this type of analysis aren’t available in any of the standard enterprise packages so implementing projects of this sort is hard – often impossible.
- This type of visitor analysis doesn’t necessarily facilitate integration with the process – and it does so only with additional “functional” insights. You still need to map what you expect the site to accomplish for any given type of visitor.
Let me start with the first point – which I think is the most important. The types of visitor segments that really matter to a business are almost always completely distinct to that business. The segments that emerged on a travel site will bear absolutely no resemblance to the segments that emerge on a financial services site or a publishing site. Nor are there good formalizations for how to find the really good variables to underscore a segmentation or how to weight them appropriately or how describe them. Yet each of these is a critical step in determining the quality of the analysis. My belief is that a method that only rewards excellent practitioners and can’t be reasonably systematized is fine – but it isn’t helpful to others to talk about how great you are at such a system. That’s just marketing. Which doesn’t mean it isn’t true, it just isn’t helpful.
Few sites are as rich in segments as an online travel service. That’s a blessing and a curse. We had almost too many segments to deal with. On the other hand, on many sites it is virtually impossible to extract behavioral cues with much meaning to the business. I know that marketing people often work with “personas” – and that these are a very popular tool with site designers. Alas, mapping personas to sites based on behavior is incredibly difficult. Using the simple rule-based techniques for segmentation available in tools like SiteCatalyst and HBX it is almost always impossible to do well. Where personas can be mapped to site behaviors, or site behaviors drive useful segments, I’m completely in favor of using them. We do this all the time. Indeed, I doubt we’ve ever done an analysis without using such cues and segments. But we have no formal method of deciding which are good and bad for a particular web site and how to evaluate whether your mapping is actually working well. That’s important, because if your behavioral cues muddy your segments, then your entire analysis will be garbage.
The third point about tools is also important. Methods that work when you can deploy any available technology aren’t necessarily useful. Most enterprises – indeed, the vast majority of our own clients – can’t take advantage of our neural segmentation tools. So building a method or approach around them just isn’t very helpful. And, here’s the deal. I don’t believe that the rule-based segmentation using primitive logic and a small subset of If-Then rules can produce good visitor segments. All of our experience with neural segmentation suggests that finding or mapping the segments thus produced was not possible with tools like SiteCatalyst, Webtrends or HBX.
Finally, I think that producing complex visitor segmentations still leaves the analyst with the dicey proposition of integrating those segments into a useful structure for doing measurement of the site. You still need to know how to map and understand the behavior of these segments against specific features on your site. And if you are deploying the conversion metrics that underscore most non-Functional analysis, you’ll only have increased your problems. Because by sub-setting your populations, you’ve further lowered the likelihood of achieving statistical significance to measuring any one site change against conversion outcomes.
To sum up, I think that visitor segmentation to identify and track personas is useful, vital and necessary. But we, at least, have been unable to give it the rigor and formal structure necessary to make it easy for other people to implement well. We have been unable to get it to work on almost every kind of site. We have been unable to do it properly in the standard tools of the day. And we, at least, have found that it still requires a method of integration into the subsequent measurement of site changes and marketing decisions.
We really do use visitor segmentation techniques a lot. They really are essential. But I believe that as valuable and necessary as these techniques are they don’t provide as useful a formal platform for understanding and doing web analytics as Functionalism. Instead, I prefer to think they provide a rich and fascinating extension to Functionalism that can greatly extend the power and utility of that basic approach.
Tag:
Add to 
Bookmark Murdok:
Gary Angel is the author of the “SEMAngel blog – Web Analytics and Search Engine Marketing practices and perspectives from a 10-year experienced guru.
