How to Assess AI Skills in Interviews [2026]

Figuring out how to assess AI skills in interviews is now a core hiring problem. Candidates list AI tools on their CVs, describe themselves as "proficient," and walk into interviews with polished talking points, but the gap between what people claim and what they can actually do is measurable and large. This post gives you a practical framework: interview questions that surface real capability, a work-sample task template, a consistent scoring method, and guidance on when a structured assessment should replace or supplement the interview entirely.

The stakes are real. According to the Microsoft and LinkedIn Work Trend Index (May 2024), 66% of leaders say they wouldn't hire someone without AI skills, and 71% would rather hire a less experienced candidate with AI skills than a more experienced one without them. If your interview process can't distinguish genuine AI fluency from surface-level familiarity, you're making expensive decisions on unreliable signal.

How Do You Assess AI Skills in a Job Interview?

You assess AI skills in a job interview by combining behavioural questions about past AI use, a live or take-home work-sample task, and a scoring rubric that maps answers to defined competency levels. Self-report alone is insufficient, you need observable evidence of how a candidate thinks with AI, not just whether they've used it.

Most interview processes treat AI skills the way they treated Excel skills in 2005: a checkbox on the job description, a vague question ("Are you comfortable with AI tools?"), and a gut-feel assessment. That approach fails for three reasons:

1. AI skill is multidimensional. Knowing how to write a prompt is different from knowing how to verify an AI's output, which is different from knowing when not to use AI. AISA's rubric covers 11 criteria across five dimensions (Prompting & Communication, Critical Thinking, Technical Understanding, Workflow & Application, and Safety & Responsibility) because a single question can't capture the full picture.

2. Fluency is contextual. A candidate who uses ChatGPT for brainstorming may have no idea how to integrate AI into a production workflow or evaluate whether an AI-generated analysis is trustworthy. You need questions and tasks that probe the specific dimension relevant to the role.

3. Confidence doesn't correlate with competence. This is the prediction gap problem, which we'll cover next.

Structure Your Interview Around Dimensions

Map your interview to the dimensions that matter for the role. A product manager needs strong Workflow & Application and Critical Thinking scores. A software engineer needs Technical Understanding and Prompting. A compliance officer needs Safety & Responsibility. Don't ask generic questions, ask dimension-specific ones.

Separate "Has Used" from "Can Use Well"

The first filter in any AI skills interview is distinguishing tool familiarity from applied competence. "Have you used Claude?" tells you nothing. "Walk me through a time you used an AI tool to solve a problem that had real consequences, what did you prompt, what did you get back, and what did you do with the output?" tells you a lot.

Why Self-Reported AI Skills Are Unreliable

Self-reported AI skills consistently overstate actual capability. Across 1,393 people who predicted their own AI fluency score before taking the AISA assessment, the average predicted score was 62.6 out of 100 while the average actual score was 44.2, an overestimation gap of 18.4 points. That's the difference between thinking you're Proficient and actually being Developing.

This prediction gap isn't uniform. It varies by role:

What the Prediction Gap Means for Hiring

Engineers overestimate by 15.1 points on average. Students overestimate by 31.2 points. If you're hiring junior engineers or recent graduates, the CV line "Proficient in AI tools" is almost certainly wrong, not because candidates are lying, but because confidence calibration in a new skill domain is genuinely hard.

This is the Dunning-Kruger effect applied to AI fluency. People who have used ChatGPT a few times feel competent. People who have built multi-step AI workflows and encountered failure modes have a more accurate (and usually more modest) self-assessment.

The CV Problem

A CV tells you what tools someone has touched. It doesn't tell you:

• Whether they can decompose a complex task into steps an AI can handle
• Whether they verify AI outputs before acting on them
• Whether they understand the privacy implications of pasting proprietary data into a third-party model
• Whether they can adapt when a model gives a confidently wrong answer

These are the skills that separate a Developing-tier user (45.4% of the 2,303 people assessed by AISA) from a Proficient one (23.2%). Your interview needs to surface them.

Which Interview Questions Reveal Real AI Skill?

The best AI skills interview questions ask candidates to narrate specific past decisions, explain their reasoning about AI limitations, and demonstrate how they handle failure modes. Avoid yes/no questions and tool-name checklists, they reward familiarity, not fluency.

Here are questions organised by the dimension they probe, drawn from the same framework AISA uses to evaluate AI fluency across its 11 criteria.

Prompting & Communication Questions

These test whether a candidate can communicate effectively with AI systems, not just write a prompt, but iterate, constrain, and steer.

• "Describe a prompt you wrote that didn't work the first time. What did you change, and why?", Tests iterative refinement. Weak candidates say "I made it more specific." Strong candidates describe what the model got wrong, what they hypothesised about why, and how they restructured the prompt.

• "If I asked you to use AI to draft a stakeholder update for a project you're working on, what information would you include in the prompt before hitting send?", Tests context engineering. You're looking for: audience, tone, constraints, what not to include, and what format the output should take.

• "How do you decide between giving an AI a single complex instruction versus breaking it into steps?", Tests understanding of prompt chaining and task decomposition.

Critical Thinking Questions

• "Tell me about a time an AI gave you an answer that looked right but wasn't. How did you catch it?", Tests hallucination detection. The best answers describe a verification process, not luck.

• "You get an AI-generated analysis with three data points you can't independently verify. What do you do?", Tests source triangulation and stakes-based review. You want to hear about checking primary sources, cross-referencing, or flagging uncertainty rather than accepting the output.

• "How do you decide how much to trust an AI's output?", Tests calibration. Strong candidates mention the type of task, the stakes, the model's known weaknesses, and whether the domain is well-represented in training data.

Workflow & Application Questions

• "Walk me through an AI-assisted workflow you've built or improved. What was the before and after?", Tests whether the candidate can integrate AI into real work, not just use it for one-off tasks.

• "You have a repetitive task that takes your team 4 hours a week. How would you evaluate whether AI could reduce that?", Tests automation ROI thinking. Look for: task analysis, error tolerance, data sensitivity, and a realistic assessment of what AI can and can't automate.

Safety & Responsibility Questions

• "What data would you never paste into ChatGPT, and why?", Tests data privacy awareness. Candidates should mention PII, proprietary code, customer data, and contractual obligations, not just "sensitive stuff."

• "Your company is considering using AI to screen job applications. What risks would you flag?", Tests bias awareness and ethical reasoning.

For candidates preparing on the other side of this process, we've published a companion guide: What AI Skills Do I Need for an Interview?.

How to Design an AI Work-Sample Task

A good AI work-sample task gives the candidate a realistic problem, access to an AI tool, and a defined output, then evaluates the process as much as the result. The task should take 30-60 minutes and mirror actual work the role requires.

Task Design Principles

1. Use a real (or realistic) problem from your domain. Don't ask candidates to "write a poem using AI." Ask them to do something the role actually involves: draft a technical spec, analyse a dataset, write test cases, summarise customer feedback, or build a prototype.

2. Provide the AI tool. Standardise the environment. If you let candidates use whatever tool they want, you're partly measuring tool access and subscription tier, not skill. Provide a shared workspace, a Claude project, a ChatGPT session, or a Cursor IDE instance.

3. Require the candidate to show their work. The conversation history is the evidence. Ask candidates to share their full prompt history, not just the final output. You're evaluating how they steered the AI, not whether the AI produced something decent.

4. Embed a trap. Include an element the AI is likely to get wrong, an ambiguous requirement, a domain-specific nuance, or a constraint the model will overlook. This tests whether the candidate catches and corrects the error.

Example Work-Sample Task: Product Manager Role

Task: You're a PM at a B2B SaaS company. A customer has requested a feature that conflicts with your current roadmap. Using the AI tool provided, draft a one-page analysis that includes: (1) a summary of the customer's request, (2) the technical trade-offs of building it now vs. later, (3) a recommendation with reasoning. You have the customer email [provided] and the current roadmap [provided]. Time limit: 45 minutes. Submit your final document and the full AI conversation history.

What you're evaluating:

Adapting the Task by Role

For engineers, the task might involve using an AI coding assistant to debug a function with a subtle logic error. For designers, it might involve generating and critiquing UI copy. For data analysts, it might involve using AI to explore a dataset and then verifying the AI's statistical claims. The AI-Native Hiring Guide covers role-specific task templates in more detail.

The key is consistency: every candidate for the same role gets the same task, the same tool, and the same time limit.

How to Score AI Skills Consistently Across Candidates

Consistent scoring requires a rubric with defined levels, anchored to observable behaviours rather than interviewer impressions. Without a rubric, two interviewers will rate the same candidate differently, and you'll make hiring decisions on noise.

Build a Scoring Rubric

AISA scores each of its 11 criteria on a 1-10 scale with defined bands: 1-2 Novice, 3-4 Developing, 5-6 Competent, 7-8 Proficient, 9-10 Expert. You can adapt this structure for your interview rubric. Here's a simplified version for a single dimension:

Publish the full rubric in /resources/the-aisa-rubric format so every interviewer on your panel uses the same definitions.

Calibrate Your Interviewers

Before you start using the rubric in live interviews:

1. Have all interviewers independently score the same 3-5 sample responses
2. Compare scores and discuss disagreements
3. Refine the rubric language until inter-rater agreement is above 80%
4. Re-calibrate quarterly as AI tools and expectations evolve

Score the Process, Not Just the Output

A candidate who produces a polished final document but whose conversation history shows they accepted three hallucinated statistics without checking is a weaker candidate than one whose final document is rougher but whose history shows systematic verification. Your rubric should weight process evidence (iteration, error-catching, judgment calls) at least as heavily as output quality.

This is where an AI skills assessment for hiring adds value even if you also run interviews. A structured assessment provides a baseline score across all dimensions before the interview, so you can focus interview time on probing the areas where the candidate's score was unexpected, high or low.

When to Use a Structured AI Skills Assessment Instead

A structured AI skills assessment outperforms interviews when you need to compare candidates at scale, establish a baseline before training investment, or remove interviewer bias from the evaluation. Interviews are good for depth; assessments are good for breadth and consistency.

The Interview's Limits

Interviews are expensive. A 45-minute AI skills interview with two panellists costs roughly 2.25 person-hours per candidate (including prep and debrief). If you're hiring for 10 roles and interviewing 5 candidates each, that's 112 hours of interviewer time on AI skills alone.

Interviews are also inconsistent. Even with a rubric, different interviewers ask follow-up questions differently, probe different areas, and anchor on different signals. The prediction gap data above shows that even candidates themselves can't accurately assess their own skills, interviewers, who have less information, face the same problem.

When Assessment Beats Interview

The Combined Approach

The strongest hiring processes use both. Run a structured AI skills assessment for hiring early in the pipeline to establish each candidate's baseline across all five dimensions. Then use the interview to go deeper: ask about the dimensions where the candidate scored unexpectedly, explore their reasoning on the work-sample task, and assess cultural fit around AI adoption.

AISA's assessment data shows that 69.1% of people assessed fall into the Emerging or Developing tiers (scoring below 60 out of 100). Only 7.6% reach Advanced or Expert. If your role genuinely requires Proficient-level AI skills, a pre-interview assessment saves you from spending interview hours on candidates who aren't close.

In AISA's own research, 80.4% of people starting the assessment agreed that AI skills should be a factor in hiring decisions (57.6% "definitely yes," 22.8% "probably yes," from 92 respondents surveyed 7-12 September 2026). The expectation is already there, from candidates as well as hiring managers. The question isn't whether to assess AI skills, but how rigorously. Read the full breakdown at Should AI Skills Matter in Hiring?.

What About AI Skills Assessments That Use Multiple Choice?

Multiple-choice tests measure recognition, not production. A candidate can identify the correct definition of "hallucination" without ever having caught one in practice. AISA's conversational format (where a separate AI evaluator scores the candidate's own words against an evidence-linked rubric) addresses this directly. For a deeper comparison of assessment formats, see Beyond Multiple Choice.

The MCP protocol's 97 million installs and the pace of new model releases (Gemini 4 Argon launched just this month) mean the specific tools candidates know will keep changing. What doesn't change is the underlying fluency: can they prompt effectively, think critically about outputs, understand how models work, integrate AI into workflows, and use it responsibly? That's what your interview (or your assessment) should measure.

---

Related reading: AI Skills Assessment Accuracy [Wingie Case Study], how one company validated assessment scores against on-the-job AI performance.

Related reading: AI Training for Employees: Measure First [2026], why baselining AI skills before training investment changes the ROI conversation.

Related reading: AI Adoption Metrics: Why Logins Mislead [2026], the difference between AI tool access and AI skill, and why usage data alone won't tell you what you need.

Frequently Asked Questions

What questions test AI skills in an interview?

The most effective AI skills interview questions ask candidates to narrate specific past experiences with AI tools, what they prompted, what went wrong, and how they adapted. Questions like "Describe a time an AI gave you an incorrect answer, how did you catch it?" and "Walk me through an AI workflow you've built" test applied capability rather than tool familiarity. Pair behavioural questions with a work-sample task where the candidate uses AI in real time and submits their full conversation history.

How do you verify a candidate's AI skills?

You verify AI skills by observing the candidate using AI, not by asking them to describe their skill level. Work-sample tasks with shared conversation histories provide direct evidence. Structured assessments like AISA provide a scored baseline across multiple dimensions before the interview. AISA data shows an average 18.4-point gap between predicted and actual scores (n = 1,393), which means self-report is not a reliable verification method.

Should AI skills be part of hiring decisions?

Yes, for most knowledge-work roles. The Microsoft and LinkedIn Work Trend Index (May 2024) found that 71% of leaders would rather hire a less experienced candidate with AI skills than a more experienced one without them. AISA's own research found 80.4% of respondents agreed AI skills should factor into hiring. The key is assessing AI skills with the same rigour you apply to other core competencies, with structured questions, observable evidence, and a consistent scoring rubric.