12 Competencies of UX Design No. 4: Measuring Audience Experience Success

Measuring audience experience success means collecting the right mix of qualitative and quantitative data -- timed correctly, segmented by user context, and adapted over time to avoid research fatigue. Done well, it tells you not just what users are doing, but what they're thinking and feeling, which is what actually drives product decisions.

What Is Measuring Audience Experience Success?

In the late eighties, psychological researcher Dr. John Gottman set out to determine whether he could tell happy couples from unhappy ones by analyzing their behavioral patterns. In a study that would eventually be dubbed the Love Lab, he invited couples to discuss a particularly contentious topic on camera. He coded behaviors like eye rolls, sighs, and tone of voice, then combined that with quantitative data -- heart rate, skin conductivity, blood pressure -- to map stress levels.

After analyzing hundreds of couples over three years, Gottman could predict which relationships would remain intact with 83% accuracy.

Gottman isn't a designer, but his methodology isn't all that different from how savvy UX professionals approach user research. Like Gottman, UX designers have to analyze not just what someone is saying and doing, but what they're thinking and feeling as well. This is what we at Drawbackwards call Measuring Audience Experience Success.

What Are the Three Elements of the Right Measurement Mix?

There are a number of research methods UX designers use to understand user experience before, during, and after an interaction -- and not all of them make sense for every situation. More often than not, successful audience experience measurement comes down to the right combination of a few methods, not all happening at the same time with the same audience. This is especially true when you're dealing with a diverse audience with complex needs.

No matter which methods you use, your strategy should be guided by three critical factors.

How Does Timing Affect the Quality of User Research Data?

Getting quality data from a specific interaction often comes down to when you ask. Ask too early, and users are more likely to shrug their shoulders than provide meaningful feedback. Too late, and they won't remember enough to be useful. Perhaps worst of all is asking in the middle of an interaction -- something that often undermines the very experience you're trying to measure.

Generally speaking, the sweet spot is right after a key interaction. That way you don't interrupt the experience, but a user's thoughts and feelings about that interaction are still fresh in their mind.

Why Does User Context Matter for Audience Research?

Just like understanding a user's state of mind and current situation can guide initial design, it can also influence how a particular experience is measured. New users and established users won't provide the same type of feedback. Rather than throwing an audience into a single testing bucket, designers need to identify what kinds of insights they can get from different types of users, and how those users change over time.

Consider grouping your audience into buckets based on things like experience level, professional role, buying phase, or average web usage. This removes tricky outliers that can throw off your findings and helps you hone in on patterns that really matter.

How Do You Prevent Research Fatigue Without Losing Data Quality?

Nobody likes getting asked the same questions over and over again. Of course, sometimes getting quality data requires asking the same users for feedback multiple times over a given period or within a specific workflow. Simply changing the way questions are phrased can be a great way to fight research fatigue. The same goes for developing new questions based on users' evolving needs.

What Does a Strong Audience Measurement Strategy Look Like in Practice?

Some companies do a great job of balancing timing, context, and adaptivity in their audience measurement strategy. Cloud-based productivity tool Asana, for example, has a robust user research strategy that accounts for all three factors in equal measure.

Before Asana users are included in their research, they must first fill out an extensive form covering everything from how much they use Asana to which industry they work in (context). From there, users are contacted to participate in either an in-person test or one that takes place over the phone, depending on their preference (adaptivity). Users that do participate take part in a carefully moderated study that gauges their experience in real-time (timing).

Not every company has the resources to conduct research this involved, but even very simple methods can be enormously successful when they account for these three factors.

Why Does Audience Experience Matter?

When Gottman created his relationship study, his goal was to determine how current behavior patterns would dictate future decisions. Measuring audience experience works the same way. For Gottman, the question was whether a couple would split -- for designers, it's whether a user will continue to use a product or service.

Just like some of the couples Gottman studied, digital audiences today have plenty of options and are constantly evaluating them to determine which they prefer. Since digital audiences sometimes have thousands of options to choose from, the quality of their experience is a major deciding factor in which they select.

To make sense of where any given experience stands, the team at Drawbackwards uses the Experience Success Ladder, which codifies levels of user experience into five distinct rungs, each building on the one below it. The higher an experience climbs on the ladder, the more likely users are to develop loyalty and engagement with it.

What Are the Five Rungs of the Experience Success Ladder?

Functional

Functional experiences fulfill technical specifications but don't include much in the way of clear labels, logical order, or user-friendliness. Settling for an experience that is merely "functional" is a great way to alienate the majority of human audiences looking to move through a workflow and achieve something easily. It's also where human error is a frequent result.

Usable

Usable experiences are different from functional ones because a user can typically complete a task correctly with some work. "Usable" and "usability" is too low a bar in the UX discipline today, yet many teams stop here. Merely usable experiences still make people think before acting and often leave a person searching for the right way to complete a task. This level still results in wasted interaction time, especially for new users.

Comfortable

When an experience moves from usable to comfortable, it becomes inherently more intuitive for a user. They won't need to dig around to find what they need and are able to move through a workflow with less friction -- but experiences that are merely comfortable may not leave a lasting impression.

Delightful

Move one rung up to delightful, and you are beginning to leave users feeling noticeably happy rather than just satisfied. Surprises appear that elevate and complement the core experience in unexpected and fulfilling ways. Users will enjoy the experience and might prefer to use it -- unless something truly meaningful comes along.

Meaningful

Meaningful experiences are those that users are likely to say they "can't live without." These are the product moments that lead people to tell their friends what they're missing out on. Savvy product managers and UX designers aspire to create meaningful experiences by understanding their audience's deeper emotions, needs, and wants. They also know how to use time, space, and context to deliver information, tools, and insights in a way that alters how people think and act in their world.

How Do You Turn Qualitative Data Into Actionable Insights?

Meaningful experiences don't happen by accident. They require a deep understanding of users' emotions, needs, and wants -- all of which can't be captured with quantitative data alone. The trick is understanding how you can create meaning from qualitative data, and that comes down to repetition. If you analyze enough qualitative data, patterns will start to emerge, and those patterns can tell you a lot about how and why users are behaving (or not behaving) in a particular way.

At Drawbackwards, we created a tool called UX Rings, part of an overall assessment process, that uses patterns and impressions in user feedback to "grade" particular experiences based on the five rungs of the Experience Success Ladder. This helps designers make sense of complex qualitative data in a format they can actually understand and use -- and helps them understand what changes they need to make to move from merely comfortable to meaningful.

Creating Meaningful Experiences

Creating meaningful experiences comes down to understanding and designing for human emotions, wants, and needs. Doing so means taking enough time, looking at the right metrics, and using the right tools to understand exactly what an audience is telling you.

Though the process takes considerable time and energy, neglecting to measure audience experience success is a one-way ticket to mediocrity or failure. Smart UX designers will always recognize the importance of knowing what their audience thinks and feels, and will allow those insights to guide their entire design approach.

Frequently Asked Questions

What is audience experience success in UX design? Audience experience success is a measure of how well a digital experience meets users' functional needs, emotional expectations, and deeper wants. It goes beyond task completion to capture what users are thinking and feeling throughout an interaction.

When is the best time to collect user feedback during an experience? Right after a key interaction. This avoids interrupting the experience while ensuring the user's thoughts and feelings about that moment are still fresh.

What is the Experience Success Ladder? The Experience Success Ladder is a framework developed by Drawbackwards that organizes user experience quality into five rungs: functional, usable, comfortable, delightful, and meaningful. The higher an experience ranks on the ladder, the more likely users are to develop loyalty to it.

How do you make qualitative UX data usable? By analyzing enough of it to identify repeating patterns. Tools like Drawbackwards' UX Rings use those patterns to grade experiences against the five rungs of the Experience Success Ladder, giving designers something concrete to act on.

Why is "usable" not a good enough benchmark for UX? Usable experiences still require users to think before acting and often leave them hunting for the right path through a workflow. In a market where digital audiences have thousands of alternatives, usable is rarely enough to build loyalty or preference.