Image by vectorjuice on Freepik
Artificial intelligence has flipped the customer service industry on its head — and implementation is only growing.
67% of leaders are expecting to increase AI-spending over the next year. The race to inject AI into customer support workflows is officially taking off like a rocket. That makes it important to consider how to know whether your investments in AI are creating a better experience for your team and customers — or if they’re just adding spend to your budget.
Metrics to measure the outcomes of implementing AI efforts are lagging behind AI adoption, so we've put together a list of the most popular metrics for tracking the success of common customer support AI features.
Big picture considerations
The influence of AI on your agent's and customer's experience can seem arcane and murky, but understanding (and measuring) that impact isn’t impossible. There are a few important things to keep in mind at the outset:
Just like in most customer experience endeavors (and life, honestly), it's important to consider both quantitative and qualitative measures. We’ll hit on a number of quantitative metrics below, but running surveys that capture your team’s experience with a tool or performing manual quality reviews can create a more holistic understanding of your AI implementation's outcome.
The best way to ensure you're able to gauge the value of your AI tools is to integrate tagging into its workflow. Leaving behind a clear marker of which interactions have been touched by AI is one of the key steps in understanding AI’s downstream impact.
Using a number of metrics (where possible) is often best. Focusing narrowly on a single KPI can obscure other factors. For example, an AI-powered translation tool might speed up your first reply time, but if it tanks your replies-to-resolution, it’s probably not actually helping.
AI features & the key metrics to measure them
Although it might feel like there are new AI-powered customer service tools launching every day, most of these tools can be bucketed into a few different categories based upon the actual thing they do for your support team. These categories include:
Expansion
Tone-change
Translation
Summarization
Auto-tagging
Macro-suggest
Intelligent triage
Some tools — like Zendesk — do several of these things, while others are focused on just one specific task.
Let’s look at each category and how you can understand its impact on your support team’s productivity.
Expansion
What is expansion?
Expansion is a feature that takes barebones text and gives it extra detail to improve the clarity and quality of your response. Using AI to expand a customer service agent's reply is one way to speed up your team’s work, without sacrificing on the quality of interactions.
Measuring its impact
Customer Satisfaction (CSAT)
Expanded replies from your team should be better replies — clearer, and more detailed. After all, there’s no point in just adding more words to a reply for no reason. Although it’s not guaranteed, one way you may see this reflected is in your customer's satisfaction with interactions that leverage expanded communications.
By analyzing your CSAT scores on tickets where AI Expand is used versus tickets where it isn’t used, you can understand the quality of the expanded replies.
Quality Assurance Score
Again, a longer reply doesn’t necessarily mean it’s better.
Leverage your internal quality scoring to compare pure human replies with their AI-enhanced equivalents. Maintaining a stable QA score with decreases in handle time, or better yet, an increased QA score, would show the positive impact of AI.
Average Handle Time (AHT)
Expansion is primarily meant to be a time-saving feature. Instead of painstakingly writing out every detail of a reply, your agents can get things started and the AI will take care of the details.
This makes AHT a great metric for understanding the impact of Expansion on your team’s productivity. You should expect to see AHT of tickets that Expansion is used on to be noticeably less than those that AI doesn’t touch.
One-touch interactions or First contact resolution (FCR)
FCR measures customer requests that require only a single response from your team to resolve. It represents an efficient response to customers — one that's clear and packed with detail.
When it’s doing its job, AI Expansion can move the needle on this metric. It won’t be a guarantee, because the AI still needs your agent to provide the core of the reply for Expansion to work, but as your team learns to work well with Expansion, you’d expect to see FCR improve.
Tone-change
What is tone-change?
Tone-change shifts the voice of a message you'd like to send to a customer, giving you the option to make it more friendly among other outputs. Tone change is a great way to ensure warm, on-brand communications with your customers. It’s particularly helpful for teams with members whose first language may not be the one you're providing support in.
Measuring its impact
Average Handle Time (AHT)
Tone-change features aren’t primarily meant as a productivity tool — they’re more about customer connection and using a consistent brand voice. That being said, if your team members are spending loads of time trying to nail the right voice, you may also see an impact on AHT here.
Quality Assurance Score
The impact of an improved tone should be clear in the internal quality checks of your team. Dialing in your AI over time should mean a highly consistent brand voice, and you should see the “Tone” section of your QA scorecard improve.
Customer Satisfaction (CSAT)
If CSAT for AI-involved conversations trends the wrong way, it might indicate an outcome that isn't in alignment with your customer's expectation of your brand. This may be a training issue, where you’ll need to work on improving the tone of voice your AI is aiming for.
Translate
What is AI translate?
AI translation automatically captures communication and translates it. One example is the Zendesk Translation app from Swifteq, which can automatically translate every ticket your support team receives.
Most AI translation is bi-directional, meaning it can both take a customer's reply and translate it to a language your team can support and translate agent responses to the preferred language of a customer allowing you to support customers all over the world.
Measuring its impact
Response Times
AI-powered translation should give your team a significant speed boost.
Instead of opening up another tab in Google Translate and copying and pasting, they’ll have instant access to a customer's request regardless of their language. An impactful implementation of AI translation should reduce your overall handle time for foreign language tickets. You should also see improvements in first reply times.
Resolution Time
Similar to response times, no longer requiring specialized agents or manual copying and pasting into a translation tool should reduce your resolution times. If not, there may be an indication that the quality of the translation isn't quite where your team needs it (or that you need to encourage more adoption of this new feature).
Knowledge Management Capacity
While not as common of a metric, AI translation tools like Help Center Translate can automatically translate your help center articles into any language — without destroying your formatting or links. This translates into massive time savings for your support team or your knowledge management team, and you should see that reflected in the capacity it frees up for them.
Summarization
What is summarization?
AI-powered Summarization is a popular feature that several major helpdesks have launched. It recaps the conversation between your team and a customer.
It can be helpful for a new teammate to take over an existing conversation and quickly get up-to-speed on what's happened. It can also be valuable for leaders looking to review conversations at speed. It’s also useful when you’re escalating a ticket up to your product or engineering team for additional support.
Measuring its impact
Average Handle Time (AHT)
A good recap of a customer conversation should make it faster for agents to resolve tickets.
A summary accomplishes that by reducing the amount of time it takes to get up-to-speed on an existing issue. You should mostly expect to see the impact of Summarization on tickets that are touched by multiple teams or team members.
Auto-tagging
What is auto-tagging?
AI-powered auto-tagging is a feature that will apply tags to incoming conversations based on the content of the customer's message as well as metadata about that particular customer (e.g, their plan level or brand.) This is meant to replace the manual process of tagging. It’s also meant to improve upon non-AI based tagging by being able to apply contextually derived tags from the customer's message.
The Zendesk ChatGPT app from Swifteq, for example, can automatically tag every ticket your support team receives with a contact reason, so your team doesn’t have to do this manually anymore.
Measuring its impact
Contact Rate
Contact rates represent the percentage of active users in a given period who reach out for support. It measures how much support your customers need when using your product.
Well-tagged conversations are one of the best ways to capture customer sentiment at scale, and they have a unique power to illuminate product needs --- and therefore potentially reduce the contact rate of your product.
While it’s not a direct line from auto-tagging to a lower contact rate (it requires some analysis and the building of product improvements), it’s a powerful long-term impact of using AI.
Average Resolution Time
Tags are usually the thing that route tickets and kick off automated workflows.
Letting AI accurately apply tags up front means improved routing and workflows triggering sooner. You should find tickets getting into the right team’s hands faster, as well as a reduction in misrouted and mistagged tickets. The overall impact of this should be decreased average resolution time.
Macro Suggest
What is macro suggest?
Macro Suggest is an AI feature that automatically suggests context-informed macros for solving customer requests. This might mean suggesting a macro to an agent for a specific ticket. In Zendesk, macro suggestions help Zendesk admins uncover new opportunities to create shared macros that your whole team can benefit from.
It’s a helpful feature, because AI is far better at noticing similar responses across your team’s thousands of tickets than a human would be.
Measuring its impact
First & Average Response Time
Providing an immediate suggestion of the right macro has the potential to be a major speed boost for your team. Contrast the average response times of tickets that utilize macros against the new AI-powered versions to see if this feature helps speed up your team's efforts.
Macro Usage
Many teams track macro usage as an individual or team metric — a way to encourage the creation of and adoption of helpful shortcuts. Automated suggestions that are high quality can improve macro adoption across your whole support team.
Quality Assurance Score
A major attribute of many quality scorecards is the application of the correct macro to the situation. If AI-informed macro usage is helping provide the right macro, you should see the impact of it on your internal quality scores.
Intelligent Triage
What is intelligent triage?
Intelligent triage is a class of AI features that help provide context about an incoming ticket to your team. They can report on the intent of the conversation (what it believes the customer wants) and language detection and sentiment (what it believes the customer is feeling.)
These are key details that can be supplied to your team instantaneously. Zendesk estimates this can save 30-60 seconds per ticket.
Measuring its impact
First Response Time and Average Handle Time
Intelligent triage packs a ton of context on top of a customer's inquiry, straight out of the gate.
When implemented well, it should arm your team with the ability to quickly understand the issue, gauge their emotional mindset, and provide an appropriate resolution. This should reduce the time taken initially by an agent to get the first reply into your customer's hands, as well as the overall time spent handling the ticket.
Agent Interactions per Ticket
By providing the customer's intent clearly upfront, intelligent triage can reduce the back and forth required to resolve a customer issue. At the same time, this metric isn’t foolproof, because if your tool doesn't capture the right info from your customers up front, even the best AI implementation won’t be able to solve for it.
But when it works, you should expect to see your agents are consistently able to resolve more tickets with fewer touches — which is great news for your ability to scale your team.
Wrapping it up
AI is incredibly powerful, and the customer support world is scrambling to integrate its power into their workflows. But just throwing AI into your tech stack doesn't mean you're making your customer experience better — you could just be throwing money into the fire.
Use these metrics as a starting point to keep your AI implementation accountable and to make more informed buying decisions.
And if you’re looking for the easiest way to get started, you can start your free 14-day free trial of Swifteq’s Zendesk ChatGPT automation app today. From automated tagging to translation to extracting key data from your incoming conversations — it can unlock a whole realm of opportunities for your team.
Written by Thomas Hils Thomas is a 10+ year veteran of the customer support space, helping high-growth startups scale magical experiences. You can connect with him on LinkedIn. |