Online Applications and highly-interactive web sites written in full-on programming languages are becoming the norm on the web, not the rare exception. Measuring these applications is challenging for a whole host of reasons. In my last post, I talked about the inevitable trade-offs between the desire for extensive measurement and the cost-of-measurement in a SaaS environment. In today’s post, I’m going to discuss some of the ways that analyzing applications is a little different than analyzing more traditional we sites.
One of the biggest differences I see between measuring applications and measuring websites is that application developers are far more interested in measuring the efficiency of their GUI. Traditional web sites have far fewer moving pieces – and the main concern is how well each page (as a unit) functions. But applications have many more options to consider and GUI developers are almost always deeply interested in understanding what users need most to accomplish any given task.
Much of the framework I suggested for measuring applications is driven by this recognition that applications can’t be measured quite like web sites. In my framework, units-of-work replace page views as the “core” ingredient in measurement. In addition, I added the concepts of “state” and “performance” to the unit of work.
From these concepts, a fairly powerful set of GUI analysis reports fall out quite naturally. It’s easy to report on which tasks are most commonly used; which tasks are done first in an application; the order of tasks and their associations (which are done together). In addition, the analyst can easily tell which application options are used most often (from application states); how application states are changed for each task; and which application states are most likely to result in task success. Finally, the performance measurements lets application developers compare how much time each task takes in total to how much of that is human time vs. machine time. It also makes it easy to see if time is impacting task success rates.
This is all good stuff and tremendously useful for GUI designers. And as I’ve said before, this type of measurement CAN be done with current web analytics systems that lack my suggested framework; it just requires quite a bit more work setting up the measurement infrastructure and hacking the reports.
There are some other types of application analysis that fall out a little less obviously from the framework and are distinct from traditional web analysis.
Unnecessary Clicks: Part of improving any GUI is making the path to task completion as swift and short as possible. The same forces are at work in traditional web sites. In a Functional analysis, we’re always looking for unnecessary pages – things like Router pages where the vast majority of users choose a single route. But use-cases are harder to apply to the open-ended non-task based paradigm that most web sites live in. With applications, however, use-case analysis (as embodied in my units-of-work concept) is probably the core methodology. For this analysis, the goal is to understand the swiftest path through any task and then identify and count all of the types of actions that deviate from that path. By identifying common deviations, designers may be able to bake-in settings, information or functionality to a process so that the user doesn’t have to – improving the overall performance of the application.
Human Performance Curves: One of the most difficult trade-offs an application designer faces is structuring a GUI that works well for beginners and experts alike. There is almost never a perfect solution – most interfaces will tend to optimize one experience or the other. But understanding the trade-offs doesn’t guarantee that your design matches your expectations. Analyzing this is non-trivial and quite different than most web site analysis. The best way to approach the problem is with performance curves. In the framework I suggested, performance times are automatically captured for each unit-of-work. These performance times measure both human time and machine-time. For this analysis, you need to plot the performance times against the number of times the user has done the task. This will yield a curve of times by usage. In an application that draws lots of regular users, you’d probably hope to see times drop sharply on repeat usage since the application should be designed to reward regulars. For infrequently-used applications, you’d probably hope to see a much flatter curve – the GUI should focus on getting all visitors through the process and not so much on speed improvements and options for heavy users.
Onboarding Success / Time: This is an analysis closely related to Performance Curves – the idea is to focus on the success of first time users. First use is critical to any application – first impressions do matter – and converting triers to regulars is vital. But the performance of first time users can be obscured if you focus on averages; especially if you have a significant population of regular users. These regulars are likely to be so good at the application that they make every measurement look good – performance times and completion rates will all be outstanding. By focusing on only first-time user performance, you can correct against that bias and better understand potential usability issues.
Help Rates: In traditional web analytics, people rarely go look for help. Well, they don’t do it that often in applications either! But there is a traditional analysis equivalent – we often analyze the internal search rate from pages, especially router pages. This rate is nearly always fairly low. But there are significant variations by page that highlight usability issues. Pages with high-rates of internal searching often lack pointers to types of content the user is expecting. “Help” usage in applications is similar. Tracking context sensitive help by unit-of-work is a good way to identify trouble spots. Remember, it’s quite unlikely that 50% of visitors are going to look for help ANYWHERE. You’re looking for tasks where the help rate is significantly higher even if the difference is between a rate of 3% and a rate of 6%. It’s almost like counting Voice of Customer complaints by task – the evidence is anecdotal but each complaint is likely a stand-in for many other users.
Common Variations Analysis: Applications are built because they provide a user with the flexibility to try different combinations of factors. It doesn’t matter whether you are talking about configuring options on a car, selecting a life-insurance policy, designing a portfolio, or choosing a vacation – the underlying goal of a good application is to make it as easy as possible for users to explore the options that might interest them. Out of any set of options, however, there are bound to be strong patterns and relationships between individual options that the application designer might take advantage of to make the application more intuitive and powerful. If you know which combinations users tend to explore in sequence or together, the application can be designed to automatically create comparisons or at least to suggest and short-cut navigation to these associated options. In my measurement framework, applications states would be used to capture this type of information. But the same analysis can be done within current measurement frameworks though you’ll probably have to export the data to a SQL-Database or statistical package after capturing use states in custom variables.
There are, of course, an unlimited number of different and often highly specific analys
is methods for applications. But just as certain methods of analysis have proven routinely fruitful for web sites (functional analysis, real-estate analysis, funnel analysis, etc.), these analysis methods fit many a different application. People often assume that having a good tool is the same as having good measurement. But as I’ve said repeatedly in this series, tight integration of measurement into an online application development environment is just a piece of the solution.
I’ve written this series as a sort of open-letter about what I’d like to see companies like Adobe, Google and Microsoft do to integrate measurement with online applications. But along the way, I’ve tried to point out how most of what I’m recommending can be done in today’s measurement system. It takes a lot more work than it should, but it’s still quite possible. Since nobody ever did useful work by waiting for tools to improve, I want to close by emphasizing this last point. People are already building application and dynamic web sites. Nobody’s waiting for good measurement paradigms to emerge and be incorporated into their tools.
By getting involved with the application development cycle early on, understanding what kinds of concepts are important to application measurement, doing your best to make your measurement infrastructure clean and testable, and adapting your analysis plan to meet the needs of applications, you can bring measurement value to the table right now.