Rewrite Your Customer Service Skills Test for the AI Era

Nearly a third of hiring teams are still figuring out how to evaluate AI skills, and that number doubled in a single year.

60% of hiring professionals say AI is making it harder to evaluate candidates’ real skills, while the share still working out how to assess AI skills jumped from 16% in 2025 to 32% in 2026.

If you hire for customer support, you’ve probably felt this already. Any take-home assignment you send a candidate can now be answered by AI in a matter of seconds, and the answer will usually read at least somewhat acceptable.

So stop trying to catch candidates using AI. Assume AI writes the first draft of every answer, and build your hiring process around what the candidate does with that draft once it exists.

Do they notice when it’s wrong? Can they make it sound like it came from someone who actually cares about the customer?

At Hospitable, we redesigned our hiring process around this idea. Below, I’ll walk through the skills we think are worth scoring and the formats we used to test them, followed by a set of customer service interview questions you can use alongside your take-home assignment.

The job you’re hiring for has already changed

Think about what a support specialist’s day actually looks like in 2026.

AI agents now handle a big chunk of the simple, repetitive questions that used to fill up the queue. Globally, AI resolved 30% of service cases in 2025, and service teams expect that to reach 50% by 2027.

That doesn’t mean there’s less work for humans.

Salesforce also found that reps using AI spend 20% less time on routine cases, and reps who work alongside AI agents now spend a quarter of their week on the most complex issues.

Here’s what tends to fill that time instead:

• Reviewing what the AI agent tells customers, and catching where it went wrong (including when it should have handed the conversation over to a human).
• Training the AI agent by spotting gaps in its answers and improving the knowledge it draws on.
• Editing AI-drafted replies before they go out. Tools like Agent Co-writer draft a reply inside Zendesk using your help center and writing guidelines, and the agent checks that it’s accurate and sounds right before sending it.
• Handling the hardest conversations, which are usually the ones the AI hands over. This leaves human support teams with some of the most emotionally loaded work in the business.

In other words, much less of the job is about writing answers from scratch, and much more of it is about judging answers that someone (or something) else wrote. If you think of AI as the writer, the person you’re hiring is a lot closer to an editor.

That’s why a customer service skills test that bans AI ends up measuring a job that doesn’t really exist anymore. It also hides a big chunk of what candidates can do, because many of the skills that matter most in the role only show up once AI is part of the work.

The four skills an AI skills assessment should measure

When people talk about an AI skills assessment, they usually mean prompting. Writing a good prompt matters, but in support it’s often the easier half of the work. The harder half is deciding what to do with whatever comes back.

The most useful lens I’ve found for this is Ines van Dijk’s Conversational Integrity Model, which she wrote about on this blog in Hiring for the Queue AI Is Building for You. It breaks support competency into three layers:

• The Mechanical Layer covers Clarity, Relevance, Informativeness, and Truthfulness: what gets communicated, and how accurately it lands.
• The Relational Layer is about restoring cooperation after something has gone wrong, for example by acknowledging the specific damage and matching the customer’s emotional register.
• The Integrity Standard includes systemic awareness, discretionary judgment, and relational responsibility. Van Dijk describes it as the engine of the whole system.

The model says that a practitioner who performs the other layers without the Integrity Standard produces responses that are “formally adequate and experientially hollow.” I haven’t come across a more accurate description of a raw AI draft.

The model is also useful for figuring out where to focus.

In her research on the automation boundary, van Dijk argues that the boundary of what AI can handle in the mechanical layer keeps moving, while the relational and integrity layers depend on conditions that better models won’t fix.

Put simply, you can expect AI to keep improving at communicating hard facts. It might even get better at simulating the type of connection and empathy expressed in the relational layer. But the integrity layer is where a human hire makes the biggest difference.

Based on that, here are the four skills I’d score in your take-home assignment.

1. Critical thinking and verification

AI drafts tend to look accurate and well-organized at first glance, which is exactly what makes them risky. Truthfulness failures are often sins of omission, where nothing false gets said but something important gets left out. Relevance failures happen when the reply answers the question the customer asked instead of the one they meant.

You want to see whether a candidate checks claims against the source and notices what’s missing.

2. Relational judgment

This means reading the customer’s emotional register correctly and acknowledging the specific thing that went wrong, rather than reaching for a stock apology. It also covers discretionary judgment: knowing when to follow a policy to the letter and when it makes sense to bend it.

AI will (or at least should) apply the policy every single time. A good support specialist knows when that’s the wrong call.

3. Tone of voice

A reply can be completely correct and still sound like it came out of a template. This is the ability to take that reply and make it sound like a real person who cares about the customer.

AI suffers from long, flowery apologies that ultimately sound insincere. A real person can cut through that.

4. Systemic awareness

Strong candidates see a ticket as a symptom of something bigger. Maybe it points to a knowledge base article that’s gone stale, or it explains why the AI agent got the answer wrong in the first place. This skill also includes owning the customer’s actual outcome, rather than just closing the ticket correctly.

Systemic awareness is the one skill that will elevate your customer experience over time.

Four ways to test AI skills in your customer service assessment test

Once you know which skills you’re looking for, the next step is designing a customer service assessment test that actually brings them to the surface.

Here are the four example formats:

• QA real AI agent responses.
• Give candidates an AI tool and review the transcript.
• Build a take-home assignment that AI will take at face value.
• Run a live working session.

1. QA real AI agent responses

Give candidates a handful of real interactions, each showing the customer’s question and the reply your AI agent sent. Then ask candidates to judge how correct each reply was and explain what they would have improved.

This is most effective if you deliberately pick interactions where AI typically struggles. Some might be replies written with total confidence that send the customer down the wrong path. Others might be technically accurate but so vague, or so padded with fluff that they don’t sound genuine.

You can usually tell candidates apart pretty quickly with this format. Strong candidates catch the confident mistake and rewrite the vague answer into something specific, while weaker ones tidy up the grammar and call it done.

It’s also a skill that keeps paying off long after the hire. Your new team member will use it every time they review an AI draft before sending it, and it’s essential if they’re going to help train an AI agent, because you can’t improve an agent’s answers if you can’t see what’s wrong with them.

(If you want to build this muscle across your whole team, Swifteq’s guide to auto QA in Zendesk is a good next step.)

Best for testing: critical thinking, tone of voice, and relational responsibility.

2. Give candidates an AI tool and review the transcript

Another option is to let candidates use an AI tool connected to your documentation, somewhere you can see every exchange.

If you have an internal knowledge AI agent, that’s the easiest place to start. If you don’t, a custom GPT or a Claude project loaded with your help docs works just as well, as long as candidates share the chat link when they’re done.

Either way, think carefully about what any AI tool in your hiring process can access. Running through an AI data privacy checklist before you give candidates access is a good idea.

This format works best when the questions in your assignment are complex and close to what working with your product is actually like. If candidates need to do some research to answer them, AI becomes a natural place to start brainstorming, and that’s exactly what you want to observe.

The strongest transcripts show candidates bringing their own thinking to the AI. One candidate who went through our process at Hospitable did their own research first, read through our documentation, and came up with a theory about what was causing the customer’s problem. Only then did they bring it to the AI:

“Here’s what the customer described. I think this might be the cause. Does that sound realistic? What are the arguments against it?”

That’s someone who shows up with their own opinion and uses AI to poke holes in it.

When you review a transcript, a few things are worth paying attention to:

• How much input they give the AI. Do they share their own thinking, or just paste in the customer’s question?
• Whether they ask for sources, and whether they actually check them.
• Whether they ever push back on what the AI tells them.
• Whether they copy and paste. Asking “Can you just generate a reply to this?” and pasting the result straight back is a red flag.

Best for testing: critical thinking and systemic awareness.

3. Build a take-home assignment that AI will take at face value

AI tends to accept whatever you give it at face value, and it’s rarely critical of the input unless you ask it to be. You can use that to your advantage.

One way to do this is to build your assignment around a report or case study with some inaccuracies built in, like sections that contradict each other or conclusions that need a much deeper understanding of the topic than the report itself provides.

A careful reader picks up on this. If you feed the same report to AI, though, the output is almost always surface-level and generic.

To calibrate the assignment, run it through ChatGPT or Claude yourself before you send it out. If the AI catches every problem, they’re probably too obvious. If you can barely spot them yourself, they’re probably too subtle. You’re aiming for something in between that a thoughtful candidate will notice.

Why does this matter so much? Because it targets the biggest problem seen with people using AI in support: they prompt before they understand what’s going on. Without that understanding, there’s no way to quality-check the answer or make it sound like it came from them.

Best for testing: critical thinking and discretionary judgment.

4. Run a live working session

You can also schedule a live session, give the candidate a task, and make it clear that you expect them to use AI to solve it.

Ask them to think out loud as they go. Why are they prompting the AI that way? What are they checking, and why do they trust (or not trust) what came back?

Seeing the reasoning live tells you a lot more than the finished answer does. It also works well as a follow-up to any of the formats above, since a take-home followed by a 30-minute walkthrough is very hard to fake.

If your team uses Zendesk, you can make this session even closer to the real job by running it in Agent Co-writer.

• Give the candidate a real ticket and let Co-writer generate a draft that pulls from your help center and follows your team’s writing guidelines.
• Then ask them to edit it until they’d be comfortable sending it, talking you through each change as they go.

That’s the exact workflow they’ll be using every day once they join, so you get a realistic preview of how they’ll handle tickets in their first week.

Best for testing: all four skills.

What about the moments when there’s no time to ask AI?

Not all support work leaves room to brainstorm with AI first. Live chat and phone support in particular demand a lot from people on the spot.

If your team works in live channels, AI simulations are worth adding to your hiring process. You design an AI agent that plays the customer and have the candidate handle the conversation in real time. You can make it more forgiving or more challenging depending on the role, and you can put a timer on it.

Because the conversation moves quickly, candidates don’t have time to run every response past AI before replying, so you get to see how they think on their feet.

To be clear, this tests how someone works without AI, so it isn’t an AI skills test. For live roles, though, that baseline matters just as much.

Customer service interview questions for hiring in the AI era

Your take-home assignment shows you the quality of someone’s work, and the interview is your chance to understand the thinking behind it.

The customer service interview questions below go beyond the assignment itself.

AI in their day-to-day work

• “Walk me through how you use AI in your current role. What do you never hand over to it?”
• “Tell me about a time an AI tool gave you an answer that sounded right but wasn’t. How did you catch it?”
• “What do you think humans should still own in support, even with strong AI tools?”

Verification and critical thinking

• “A customer says our help center told them one thing, and you suspect the article is out of date. What do you do?”
• “How do you decide when an answer needs checking with another team before you send it?”

Relational judgment

• “A customer is frustrated because our AI agent sent them in circles. How would you open your reply?”
• “Tell me about a time you made an exception to a policy. How did you know it was the right call?”
• “What’s the difference between an apology that helps and one that makes things worse?”

Tone of voice

• “Here’s an AI-drafted reply. Rewrite the first two lines so they sound like you.” (This also works as a quick, two-minute customer service skills test.)
• “Based on what you’ve seen of our help center, how would you describe our tone?”

Systemic awareness and working with AI agents

• “You notice our AI agent keeps giving the same wrong answer. What do you do next, and who do you tell?”
• “If you were training our AI agent, what would you want it to stop doing first?”
• “Tell me about a pattern you spotted across tickets that led to a change outside of support.”

The take-home debrief

• “Where did you disagree with the AI, and what did you change in its draft as a result?”
• “What would you have done differently with more time?”

When you’re listening to the answers, pay attention to how specific they are. Strong candidates tend to name exactly what they checked and why, while a vague “I always double-check everything” doesn’t tell you much at all.

Hiring for judgment in the AI era

The take-home assignment still has a place in your hiring process. It just needs to test for different skills than it used to.

What you’re hiring for is a person who can tell whether an AI draft is accurate and whether it actually solves the customer’s problem.

That’s also what good AI-assisted support looks like day to day.

Building the test from your AI agent’s real conversations

The first format above, QA real AI agent responses, is only as good as the conversations you pick for it. You want the ones where your AI agent sounded sure of itself and was wrong, or answered so vaguely that the customer gave up, and finding those usually means scrolling through weeks of messaging tickets by hand. So most tests end up built on the few examples someone on the team happens to remember.

Swifteq’s MCP Server for Zendesk connects Claude or ChatGPT to your Zendesk, so you can ask for those conversations instead of hunting for them. Claude can list the messaging tickets your Zendesk AI agent resolved on its own, read each conversation, and flag the ones where the customer didn’t get an answer, like someone who asked twice for a return label and then left. Each conversation shows which messages the AI agent wrote and when it handed over to a person, which makes it easy to pick a mix of confident mistakes, vague answers, and handoffs that came too late.

Because these are real customer conversations, ask Claude to leave out names, email addresses, and order numbers when it writes up the test, and check the result against the AI data privacy checklist above before anything reaches a candidate. The MCP Server can only read your tickets until you give it write access, and Swifteq never stores ticket conversations, or any personal information (PII) from Zendesk about your customers or employees.

The same reviews keep paying off after the hire. Noticing that the AI agent keeps giving the same wrong answer, one of the interview questions above, is much easier when your new team member can ask Claude to read last week’s AI agent conversations instead of only the few that reached their queue. What they find can go to their team lead or into a fix for a stale help center article, which is the systemic awareness you hired them for.

The MCP Server is free for 500 calls a month per Zendesk account, with no credit card required. Each conversation Claude reads is one call, so reviewing 100 AI agent tickets takes about 101 calls, and the free calls cover about four reviews of that size each month. Connect Claude or ChatGPT to Zendesk and build your next hiring test from your own AI agent’s conversations.