How do we measure the success of User Experience?

The aim of a usability study is to make products more user-friendly. You can do this by gaining qualitative insights about what works (or does not).

No objective score

Spoiler: it is impossible to give an objective score to the user-friendliness of a digital product. You can only measure the perception of user-friendliness, in other words whether customers find it user-friendly. It is important to realise that people who have strong self-monitoring as a character trait often give inconsistent answers and are inclined to provide socially desirable reactions.

Since the 1980s, several questionnaires have been developed that are used to measure the (perception of) user-friendliness. They all contain different formulations and structure, but come down to the same thing: the result is an unambiguous number.

Questionnaires

The SUS (System Usability Scale) is one of the oldest and most popular questionnaires measuring usability. This is because it’s open source, it’s well known and has been translated into multiple languages. This questionnaire, with its10 statements, has been scientifically tested for validity (whether it measures what it is supposed to measure) and reliability (whether it gives consistent results when tested in multiple rounds of testing).

In 2010, the UMUX (Usability Metric for User Experience) was designed that achieves the same results as SUS with 4 statements. In 2013, a UMUX-Lite was tested that provides results using only 2 statements:

In 2010, the UMUX (Usability Metric for User Experience) was designed that achieves the same results as SUS with 4 statements. In 2013, a UMUX-Lite was tested that provides results using only 2 statements:

The satisfaction score indicates the extent to which the actual performance of the product corresponds with the user’s expectations. Starting by asking questions about expectations and then about the actual experience appears to be a good solution but this is not always convenient in practice. This might be because you want to get the client through the onboarding process as quickly as possible, or because you don't want to risk boring the customer with two questionnaires, no matter how short they are.

Disconfirmation scale

Fortunately, the process can be simplified. There is evidence that a Disconfirmation Scale can measure satisfaction just as well as the method we just described. With this method you ask the customer about their satisfaction with the product compared to their expectations beforehand.

It is long established that people can subconsciously adjust their memories afterwards so that they form a consistent image with their present thinking (this effect is called rationalisation). But even if that effect is present in the Disconfirmation scale, that is not necessarily a bad thing. The most important thing is the impression that the product ultimately left on the customer, and what they think about it.

This approach works best for measuring 'overall' satisfaction with the product.

The Customer Satisfaction Index (CSAT) is a good choice to measure product satisfaction in the 'here and now'. You can zoom in on specific activities or parts of the product - checkout flow, delivery, customer service etc.

To get realistic results you need to make sure the question is neutrally phrased, so not 'How satisfied are you with X?' but 'What do you think of X?'

The Net Promoter Score (NPS) is often used as a method of measuring satisfaction. However, the NPS is a value that measures loyalty and stated intentions and not satisfaction per se (even if there is a correlation between them).

There is also a lot of criticism about the way the calculations of the NPS are interpreted. Jared Spool, expert, researcher and speaker in the field of usability, software, and design summarises it clearly here. So there are plenty of reasons not to use NPS for this purpose.

Other parameters

It’s also useful to monitor parameters like adoption rate, retention rate and churn rate. They only provide an indirect indication of satisfaction and cannot be viewed as an isolated number: '39% adoption rate' in itself says little. Therefore, these metrics are measured on a period-by-period basis so you can see trends over time. They should be viewed in the context of the industry benchmark because they allow you to analyse whether it is getting better or worse.

Determining whether the product is relevant and matches customer needs is the holy grail of every product development process. You want to know whether what you build matters, and whether a product-market fit has been achieved. In short: 'Do we sell something that people need?'

Unfortunately, there is no silver bullet, let alone a hard number or percentage. If it were easy, 'No market need' would not be the main reason why 42% of startups fail.

To estimate as accurately as possible how relevant your product is, you should use a mix of qualitative and quantitative methods. As for quantitatively measurable options, you can use the following methods to get clues as to whether your product is relevant to the end user.

'How disappointed would you be…'

One of the ways to measure the relevance of the product is described in the book 'Hacking Growth: How Today's Fastest-Growing Companies Drive Breakout Success'. Ask your customers who have recently used the product the following question: 'How disappointed would you be if you couldn't use this product anymore?'

If more than 40% of customers indicate that they would be disappointed, you have reached product market fit (i.e. your product is useful and relevant). This approach is useful for products that are still in development and have not yet defined their Unique Selling Points.

Suppose you haven’t developed a product or don’t have a customer base. In that case, you have to make do with your own assumptions and the predictions your target group makes about their own behaviour. Answers to this type of hypothetical question are less reliable than questions about past behaviour.

Technology Acceptance Model (TAM) is used to measure, among other things, the perception of utility. You can use this to get a sense of whether there is a need for the product. It is necessary to adapt and customise the questionnaire for every product and survey based on that model.

Another more nuanced model than TAM is the Universal Theory of Acceptance and Use of Technology (UTAUT). However, this nuance means that you will also have to ask a lot more questions.

Combination of quantitative and qualitative methods required.

Because product relevance is a very complex subject, you cannot (only) measure it with a questionnaire and express it with a number. Make sure you use multiple methods, for example customer journey mapping, Jobs To Be Done interviews, different LEAN experiments (pre-selling, landing page, explainer video), etc. Multiple methods together give you a clearer overall picture and more certainty in the answers.

After you have established the foundations of your digital service it’s time to look further. How can you exceed customer expectations and make them enthusiastic about your product/organisation? Achieving customer delight, that fleeting 'extra' is your new goal.

So what exactly does 'delight' mean? That depends on your organisation. There is no single definition of delight, and it is interpreted in different ways by different organisations and people.

The question is therefore whether your product should create the feeling of 'delight'. It is not an ultimate goal of UX. For some industries you should look for a different expression or interpretation of delight. For a tax consultancy firm, it could be "solid" or "calm." For a b2b shipping company, 'reliable' or 'businesslike' is an appropriate feeling. 'Delight' can be unwarranted or even counterproductive in these kinds of industries.

List of adjectives

One of the easiest ways to measure delight is to present the customer with a number of adjectives that represent the emotions and values ​​you aspire to. You can include negative emotions in that list, and values ​​that are opposite to what you want to achieve. The customer must indicate how often they experience those emotions with the product.

In many definitions, "delight" implies a positive surprise. Once you've experienced or seen something once, it's no longer a surprise or delight, but something just that you expect. The KANO model provides further insight into how this effect works.

Using UX as a means of achieving business goals is the final step in the journey to UX maturity. This also includes measuring the effect of UX on the intended business goals.

How do you go about this?

There is no one-size-fits-all solution for this. Depending on what your company wants to achieve and which products or solutions it offers, you will measure it in different ways.

Start with the business goals. This applies to NGOs and for-profit companies. Is this growth? efficiency? Behavioural change? What role does the (digital) product play in achieving that goal? Capture it. You translate this role into specific goals, which you can then develop with the Goals-Signals-Metrics technique of the Google HEART method.

Sounds complicated? An example to clarify this. Goals that are not directly expressed in terms of conversions or earnings are more difficult to deal with, so let's discuss a fictitious product for a non-profit as an example.

Example

Suppose the Ministry of Health, Welfare and Sport releases an app that anonymously guides people when they’re quitting smoking.

What is the aim of the Ministry? What are they doing to realise this vision?

• The Netherlands is healthy and well

• One of the steps is the 'Stop smoking' app

What is the purpose of this app?

• Guiding users in successfully quitting smoking

What are the signs that users are or will be quitting smoking?

• The users start the step-by-step plan

• The users follow the step-by-step plan to stop smoking

• The users maintain the new habit and do not start smoking again

What are the digital metrics of the 'follow the step-by-step plan'?

• Total percentage of people who have indicated that they have not smoked at least 5 times in the last 7 days.

• Percentage of people who continue to check in daily halfway through the step

• Percentage of people who complete the step

Metrics of this kind need to be measured over time. Your main interest is the trend, not necessarily the absolute number. When a percentage has become higher after the UX change than before, you have reached the moment when you have a reliable measurement of the impact of the UX work. If there have been several adjustments in that period within and outside the UX area of ​​the app, the trend is of course less directly attributable to UX.

Experience metrics and metrics of the effect of UX

You see: there are no examples of conversion rate, satisfaction, task success. These are the metrics of the experience and are instrumental in achieving good experience - and indirectly the business goals. But these scores, while important, don't necessarily mean that the business goal has been achieved.

These measures are difficult to define and demand maturity from the organisation and clarity from management about what 'success' means. Conversations about this help bring the role of UX closer to the business and strategic business choices.

A score is not an insight

The methods in this article are quantitative and therefore result in a score. But as useful as they are, they only tell the 'what' of the story and must be supplemented with other research methods to find out the 'why'. With these quantitative insights, you can often measure improvements over time or compare the current score to an industry average benchmark. It is worth bearing in mind that such a score does not provide insight into which steps you should take to improve it. It is best to supplement a quantitative measurement with qualitative research that also provides answers to questions such as 'why do people think this?' or 'what do our customers expect?'

A score is just a score. It’s what you do with that is what makes the real difference.