Koru Technologies
Bansi Mehta

by BANSI MEHTA - UX Design Expert

Founder & CEO, Koru Technologies

According to Don Norman, who first coined the term, “User Experience (UX) encompasses all aspects of the end user’s interaction with the company, its services and its products.” True, user experience goes far beyond giving customers what they say they want, or providing checklist features. In order to achieve high-quality user experience in a company's offerings there must be a seamless merging of the services of multiple disciplines, including user research, engineering, marketing, graphical and industrial design and interface design.

Although UX is a universal concept, applicable to all physical and digital products and services, it is most commonly studied in its applicability towards digital products and services, like software, apps and websites.

Let me paint a picture for you – you built a website, which gets a lot of traffic. However, the conversion rate is disappointing at its best. Or a lot of people signed up for your software, but have not returned after a first few days. Or else, consider that though initial number of downloads for your app were high, thanks to the marketing and promotions, but more than half of the users never bother to use it, or even worse, have uninstalled the app within a week or two.

Driving traffic to a site, software or app is easy; deriving initial conversions can also come easy if you are ready to splurge. However, making your users stay engaged and loyal to your product goes beyond marketing. Usability and user experience becomes a crucial factor here.

Steve Krug (author of Don’t Make Me Think – highly recommended for internet dwellers) defines Usability as making sure something works well, and that a person of average ability or experience can use it for its intended purpose without getting hopelessly frustrated.

However, usability is just the tip of iceberg that is User Experience.

With the rising competition in digital market, product managers and entrepreneurs are steadily investing more time and resources in honing User Experience for the products and services they offer.

For instance – automobile industry stays centred around physical, tangible products fair and square. With minor digital integrations in smoother drive around curves, terrains and roundabouts; better music system and safety measurements, automobile giants like Audi, BMW, Ford and Volkswagen have been trying to one up each other since generations. However, with its disruptive technology of self-driven electric cars, Tesla Motors has not only ousted its competition but also created a niche market for itself. An intuitive UX and product design gives Tesla an edge over other automobile players, proving that if done right, UX can find its applicability in all segments.

What are metrics?

Metrics

Spuriously, UX is considered a subjective aspect.

UX metrics are quantifiable, measurable and reliable data points that evaluate the effectiveness of your UX strategy. They can be aligned to your marketing metrics as well as compared against a ROI to calculate financial feasibility and scale profits accordingly.

UX metrics lie under three broad categories-

1. Usability Metrics or Descriptive Metrics

As I pointed out earlier, usability is just the tip of the UX iceberg.

Usability metrics focus on the ease of task or event completion by users. It includes commonly tracked metrics like time on task, navigation vs. search, task success rate, ease-of-use rating, etc. Some advanced metrics may also include tracking of interaction pattern.

2. Engagement Metrics or Perception Metrics

Engagement metrics focus on the way users interact with a site, app or software. Apart from the time per interaction, it also focuses on event streams, page views, scrolling, and most importantly, attitude of the interaction.

3. Conversion Metrics or Outcome Metrics

Conversion metrics are just as important for the UX team as they are for marketing team. It focuses on a relatively small set of users who are ready to commit to your product via prolonged engagement, usage and retention. Insights provided by conversion metrics including conversion rate and Net Promoter Score (NPS) can also help design better solutions that help a business meets its goals.

Common Methodologies

Metrics

Various companies and UX agencies deploy various methods and data points to track UX metrics. Before I dive into which metrics are right for your product, let’s take a look at the prevalent methods and key metrics:

Method 1: Net Promoter Score (NPS) Introduced by Bain & Company founder, Fred Reichheld, Net Promoter Score (NPS) is a customer loyalty metric that gauges how willing a customer is to recommend a product or service.

At its core, NPS tracks how customers represent an organization to their friends, families and associates. The same is true for virtually any product or service.

Net Promoter Score categorises users into one of three groups: Promoters, Passives and Detractors through their response to a simple question: “How likely is it that you would recommend [Organization X] to a friend or colleague?”

Respondents are given a standard scale that ranges from 0 (not at all likely) to 10 (extremely likely). Based on the responses, users are categorised as Promoters (9-10), Passives (7-8) and Detractors (0-6).

Promoters are the users who are loyal and likely to refer others, thus helping you to lower your cost per acquisition. Passives are generally satisfied users who can sway both ways, depending on competitive offers and incentives. However, Detractors are highly unsatisfied users, who may have serious issues against your product.

Net Promoter Score is calculated by subtracting the percentage of Detractors from the percentage of Promoters. The score is generally displayed as a whole number by dropping the percent sign for the final number and scores range from anywhere between –100 (all Detractors) to +100 (all Promoters).It is imperative that in order to be successful, you would want to convert your Detractors to Promoters.

One of the main reasons why so many companies (physical as well as digital) choose Net Promoter Score is its simplicity. Although NPS lacks deep insights and specific reasons for a product’s success or failure, it can help you gain an insight on the overall scenario and the level of urgency required to handle a given situation. This is why, NPS is commonly used as an alternative or a supplement metrics to measure UX.

Method 2: System Usability Scale (SUS) Another standardised UX measurement methodology is a usability testing tool developed by John Brooke in 1986, known as System Usability Scale (SUS). Comprised of a ten-item questionnaire, with five response options for respondents (ranging from Strongly Agree to Strongly Disagree), SUS helps UX professionals gather subjective information which NPS cannot provide.

Interpreting SUS scores is a bit complex. The participant’s scores for each question are converted to a new number, added together and then multiplied by 2.5 to convert the original scores of 0-40 to 0-100. This is then used to produce a percentile ranking. Studies show that a SUS score above a 68 would be considered above average and anything below 68 is below average.

Although SUS is a very easy method to administer and can be used on small user/test-case samples; the scoring system is complex and is not diagnostic.

Method 3: PULSE Metrics Slightly more insightful methodology employed by UX professionals and agencies. This method includes tracking of the following aspects-

Page views

Number of pages viewed by a single user.

Uptime

Percentage of time the website, app or software is accessible to the users.

Latency

Amount of time it takes to transfer data from one location to other.

Seven-day Active Users

Number of users who interacted with the website, app or software in last seven days.

Earnings

Revenue generated by the website, app or software.

While the PULSE metrics still remain more insightful than NPS or SUS methodology, it should be noted that it lacks context sometimes, thus becoming redundant for UX measurement. For example, the number of Seven Day Active Users reflect at the sticky quotient of the site, app, or software, but it doesn’t differentiate between new and returning users and doesn’t provide any insights on the users’ level of commitment.

Method 4: HEART Framework & Goals-Signals-Metrics ProcessDeveloped by Kerry Roden, Hillary Hutchinson and Xin Fu of Google Research team, HEART framework fulfilled the shortcomings of PULSE metrics. Based on large-scale behavioural data, HEART framework stands for Happiness, Engagement, Adoption, Retention and Task Success. The categories can be further demarcated as –

Happiness

Tracks attitudinal metrics about a user’s feelings towards the product. It includes subjective aspects of user experience, like satisfaction, visual appeal, likelihood to recommend, and perceived ease of use. With a general, well-designed survey, you can track the progress over time, with each modification incurred in the product.

Engagement

Measures the user’s level of involvement with the product. It includes metrics like the frequency, intensity or depth of interaction over a time period. It is generally more useful to report Engagement metrics as an average per user, rather than as a total count – because an increase in the total could be a result of more users, not more usage.

Adoption

Indicates the number of new users who decided to use the product within a given period of time. For instance, the number of sign-ups, downloads and purchases made.

Retention:

Refers to the number of users who ‘adopted’ at an earlier time period are still ‘engaged’ with the product. This includes sticky quotient and retention rate measured over a time period, subscription renewal and repurchases.

Task Success

Encompasses several traditional behavioural metrics of user experience, such as efficiency (e.g. time to complete a task), effectiveness (e.g. percentage of tasks completed) and error rate (e.g. percentage of failed attempts to log in).

Each of these categories have specific metrics that can be applied for an individual feature/ section or the entire product/ project. The HEART framework can be modified as per the project requirement; by adding key metrics per category, by removing irrelevant categories or by focusing on a single category.

The Goals-Signals-Metrics Process Each metric from the HEART framework is scaled against a Goals-Signals-Metrics process using a simple dashboard. It is a simple method that keeps the entire UX team in sync through articulating the Goals of a product or feature, then identifying signals that indicate success, and finally building specific Metrics to track on a dashboard. Further details on the Goals-Signals-Metrics process are as below –

1. Goals

The first step is assigning goals of the product or the feature, in terms of user experience. It may be related to task success, or increasing engagement, or to simply attract more traffic / users.

2. Signals

Once the goals have been determined, choose measurable signals that are sensitive and specific to the goal, as well as the method to collect and map those signals.

3. Metrics

To capture metrics, transform the signals in to measureable data points, suitable for tracking over time on a dashboard. Mean, fraction and percentage rates are preferred over raw data for normalizing. It is also advised to filter out automated data and drawing comparisons with other competing features or products.

The combination of the HEART framework with the Goals-Signals-Metrics process provides you precise information and analytical solutions to improvise User Experience.

Key Metrics

Metrics

Some of the frequently measured UX metrics are –

Task Success Rate

Task success rate is the percentage of correctly completed tasks by users. It reflects how effectively users are able to complete certain tasks.

Time on Task

Time spent on each task provides us insights on how much time is being spent on successful completion of a pre-defined task. Prolonged processes affect the UX negatively.

User Errors

During a given task, the number of error opportunities provide error rate. Depending on the frequency and type of errors, we can evaluate the UX and usability of a website, app or software.

Search vs. Navigation

Analytic tools provide much required information about users’ interaction with a site, software or app. Navigation vs. Search Ratio can provide information about the pages / features most frequently used.

Form Usage

Every website, app or software uses a form to capture users’ information. However, from UX point of view, it is advised to keep the forms restricted to essential fields only. Measuring the time taken to fill up a form or the number of visitors who abandon the process can provide an idea about the situation.

Back Button Usage

Again analytic tools can tell you how frequently the ‘back’ button has been used while navigating through your website, app or software. A higher number indicates a broken or faulty information architecture.

Random Visitors vs. Conversions

While high traffic is a good thing, conversion rate is the most important metric for any website, app or software.

Sticky Quotient

Apart from conversions, sticky quotient of a website, software or app also mark the effectiveness of UX and usability.

Offline vs. Online Visits

In case you own an online as well as an offline entity, the difference between the number of visitors as well as conversions you get on both mediums should be a fair indicator of your UX and usability on digital platforms.

Implementation

Metrics

Now that we are well acquainted with the commonly tracked metrics and methodology to evaluate the success of a UX strategy, let’s figure out a plan for implementation.

For a perfect evaluation of UX metrics, it is important to develop a deep understanding of define your organisation’s Key Performance Indicators (KPIs) as well as your target audience / users.

Once you have determined KPIs, link them to UX metrics to create a direct relativity between business goals and customer satisfaction. This can offer some simple solutions to improve user experience.

After you start tracking particular UX metrics in correlation to your business KPIs, make sure that you repeat calculations over a period of time to observe trends and get better idea of what’s working and what is erroneous.

Moving further ahead, you would like to work on benchmarking the key aspects, features and tasks to improve overall user experience. With a fair idea of important tasks, features and aspects of your product, you can begin on improvisation plan. Frequent iterations, A/B and multivariate testing should reflect the context of each modification implemented in terms of UX as well as its corresponding KPI. The entire process keeps your UX strategy on track and in alignment to business goals.

It should be also noted that measuring UX is an ongoing process. Periodic UX audits should be performed to evaluate right methods and metrics. With a stolid plan of action and fair evaluation of UX strategy using the UX metrics, your products can keep the users happy and scale higher profits.