AI Overconfidence by Role: Who's Worst? [Data]

AI overconfidence isn't evenly distributed. When 1,596 professionals predicted their AI fluency scores before taking AISA, every single role group overestimated — but the size of the gap varied by a factor of four. Students overshot by 31.5 points. Founders missed by just 7.1. That spread tells us something important about how different professional contexts shape confidence calibration around AI skills.

This isn't the same story as our dimension-level analysis, which looks at which skill areas people misjudge. And it's distinct from our student-specific deep dive. This post is about roles — which professional identities carry the biggest blind spots, and what drives the variation.

The Overall Prediction Gap: 17.8 Points of Overconfidence

Across 1,596 AISA candidates who submitted a self-predicted score before their assessment, the average prediction was 62.3 — solidly in the Proficient tier. The average actual score was 44.5 — squarely in Developing. That's a 17.8-point gap, meaning the typical person believes they're roughly a full tier above where they actually land.

This pattern echoes what psychologists David Dunning and Justin Kruger documented in their original 1999 research: people with limited competence in a domain tend to overestimate their ability because they lack the meta-cognitive skills to recognise what they don't know. The effect has been replicated across dozens of domains, and AI fluency appears to be no exception.

What 17.8 Points Actually Means

To put this in context, 17.8 points is the difference between a Developing score (28-59) and a Proficient score (60-79). It's the difference between someone who can use ChatGPT for basic tasks and someone who can decompose complex workflows, evaluate outputs critically, and select appropriate tools for specific problems.

When someone predicts 62.3 and scores 44.5, they're not slightly off. They're operating with a fundamentally different mental model of their own capability.

Why Self-Assessment Fails for AI Skills

AI fluency is particularly susceptible to overconfidence because the feedback loops are weak. If you write bad code, the compiler tells you. If you write a bad prompt, the model still gives you something — often something that looks plausible. Without structured evaluation, there's no signal that your output could be significantly better. This is why source triangulation and verification habits matter so much, and why most people don't develop them without deliberate practice.

A 2024 study from Stanford's Institute for Human-Centered AI found that 64% of workers who described themselves as "proficient" with generative AI tools could not correctly identify a hallucinated output when presented with one. The gap between perceived and actual skill isn't just an abstract measurement problem — it has real consequences for output quality.

AI Overconfidence by Role: The Full Ranking

The aggregate gap is interesting. The role-by-role breakdown is where it gets actionable. Here's every role group ranked from most overconfident to most calibrated:

Three things jump out immediately.

The 4x Spread

The gap between the most overconfident group (Students, 31.5 points) and the most calibrated (Founders, 7.1 points) is enormous. This isn't a case of everyone being slightly off — different professional contexts produce radically different levels of self-awareness about AI capability.

Predictions Cluster, Actuals Don't

Look at the predicted scores: they range from 60.3 to 69.1 — a spread of less than 9 points. Everyone thinks they're roughly Proficient. But actual scores range from 35.2 to 54.3 — a spread of 19.1 points. The predictions are compressed; reality is dispersed. People converge on a similar self-image regardless of how different their actual skills are.

Technical Identity ≠ Calibration

Engineers and Data professionals — roles that might be expected to have better calibration given their quantitative training — still overestimate by 15.0 and 18.1 points respectively. Technical identity doesn't automatically produce accurate self-assessment in a new domain.

Why Students and Data Roles Overestimate Most

Students overestimate their AI fluency by 31.5 points — nearly double the overall average gap. Data professionals overestimate by 18.1 points. These two groups share structural reasons for their miscalibration, but the mechanisms differ.

The Student Confidence Problem

Students predicted an average score of 66.7 (Proficient) but scored 35.2 (Developing). Their actual scores are the lowest of any role group in the dataset, yet their predictions are the second-highest. This is the Dunning-Kruger effect in its purest form.

Several factors compound the problem. Students are often early in their AI journey, meaning they've been exposed to AI tools but haven't yet encountered the boundaries of their knowledge. They may have used ChatGPT extensively for coursework without ever needing to verify outputs, evaluate model limitations, or think about responsible deployment. Usage frequency gets conflated with competence.

Our detailed student analysis shows that students score lowest in Technical Understanding (30.9) and Safety (30.9) — the two dimensions that require understanding how AI works and when it fails, not just that it works. These are precisely the areas where overconfidence is most dangerous, because errors of understanding compound silently.

McKinsey's 2024 Global Survey on AI found that organisations hiring recent graduates for AI-adjacent roles reported a 40% higher rate of "AI-related quality incidents" compared to teams staffed with experienced professionals — a finding consistent with the calibration gap we observe.

The Data Professional Paradox

Data professionals present a different puzzle. They predicted 67.6 and scored 49.5 — an 18.1-point gap. Their actual scores are middling (Developing tier), but their predictions are the highest of any group.

The likely explanation is adjacent expertise bias. Data professionals work with statistical models, understand concepts like training data and evaluation metrics, and may assume this knowledge transfers directly to AI fluency. But AISA measures practical AI fluency across five dimensions — including prompting skill, workflow integration, and safety awareness — not just technical understanding. A data scientist who understands transformer architecture but can't construct an effective multi-step prompt or identify when to keep a human in the loop will score lower than they expect.

Their Technical Understanding score (42.4) is notably lower than Engineers (50.8), suggesting that even the domain knowledge transfer isn't as strong as assumed.

Why Founders and Product Managers Are Most Calibrated

Founders overestimate by 7.1 points. Product managers by 9.7. These are the two most calibrated groups in the dataset — and the pattern isn't coincidental.

The Founder Advantage

Founders predicted 61.4 and scored 54.3. Their gap is less than half the overall average. They also have the highest actual composite score of any role group in our prediction dataset.

Founders tend to be generalists who've been forced to evaluate AI tools across multiple business functions — marketing, engineering, operations, customer support. This breadth of exposure creates more accurate mental models because they've seen AI succeed and fail across different contexts. They've had to make buy-vs-build decisions, evaluate vendor claims, and explain AI capabilities to investors and customers. Each of these activities forces a reality check.

Our founders vs engineers comparison explores this dynamic in more detail. Founders score particularly well on Critical Thinking (50.2) and Safety (50.4) — the dimensions most associated with calibration and judgment.

Product Managers: Built-In Feedback Loops

Product managers predicted 63.2 and scored 53.5 — a 9.7-point gap. As our product manager prediction analysis details, PMs benefit from a professional habit that directly improves calibration: they routinely scope what's feasible, test assumptions against user data, and adjust plans when reality diverges from expectations.

This estimation-and-correction cycle is essentially calibration training. PMs who've shipped AI-powered features have direct experience with the gap between what AI could do in a demo and what it actually does in production. That experience deflates predictions toward reality.

Product managers also score highest on Workflow & Application (57.2) — the dimension that measures practical integration of AI into real work processes. They're not just theorising about AI; they're using it in structured, goal-oriented ways that expose its limitations.

What AI Overconfidence Costs in Practice

Overconfidence about AI skills isn't just an ego problem. It creates measurable costs for individuals and organisations.

For Individuals: Stalled Growth

When someone believes they're Proficient (60-79) but actually scores as Developing (28-59), they don't seek out the learning they need. They skip foundational skill-building because they think they've already passed that stage. The result is a plateau — they keep using AI the same way, getting the same mediocre results, and never understanding why their outputs don't match what they see from more skilled colleagues.

This is particularly acute for career changers, who may be investing in AI skills as a differentiator. If their self-assessment is off by 18+ points, they're building their career strategy on a false foundation.

For Teams: Misallocated Training

Engineering managers who rely on self-reported AI skill levels to plan training will systematically under-invest in the people who need it most. If your engineering team self-reports as Proficient but actually scores as Developing, you'll skip the critical thinking and prompting fundamentals training that would produce the biggest gains.

The prediction gap data suggests that role-specific training strategies are essential. A one-size-fits-all AI upskilling programme will miss the mark differently for each group:

• Students need foundational calibration — they don't know what they don't know
• Data professionals need to bridge the gap between statistical knowledge and practical AI fluency
• Engineers need to recognise that coding ability doesn't automatically transfer to AI skill
• Designers need structured exposure to AI capabilities beyond image generation
• Product and Founders need advanced skill-building, since their calibration is already reasonable

For Hiring: False Signals

When candidates self-assess AI skills on a resume or in an interview, the signal quality varies dramatically by role. A founder who claims to be "proficient with AI" is probably close to accurate. A student or career changer making the same claim is likely off by 20-30 points. Without an evidence-based assessment, hiring managers are flying blind — and the blindness isn't uniform.

Gartner's 2025 HR Technology survey reported that 76% of organisations still rely on self-reported skill levels for AI capability planning. Given the prediction gaps we observe, this means most workforce AI strategies are built on systematically inflated data.

How to Close Your Personal Prediction Gap

The good news: overconfidence is fixable. The prediction gap isn't a personality trait — it's an information deficit. Here's how to close it.

Step 1: Get an Evidence-Based Baseline

Self-assessment doesn't work for AI skills. The data is unambiguous on this point. You need an external measurement that evaluates what you can actually do, not what you think you can do.

AISA's assessment uses a conversational format where an AI facilitator explores your skills across 11 criteria, and a separate AI evaluator scores your responses based on evidence in your own words. The scoring is independent of the conversation — you can't talk your way to a higher score. Every assessment produces a free report with your composite score, dimension breakdown, and a persona that captures your AI usage pattern. AISA's test-retest reliability shows 98% of retakers land in the same or adjacent tier, so the baseline you get is stable.

Step 2: Identify Your Weakest Dimension

The prediction gap isn't uniform across dimensions. Most people overestimate most in Technical Understanding and Safety — the areas where cognitive surrender is easiest because the feedback is weakest. Your AISA report breaks down scores across all five dimensions, so you can see exactly where your self-image diverges most from reality.

For context, here are the dimension averages across all 2,506 AISA candidates:

Technical Understanding is the lowest-scoring dimension overall, and it's also where the gap between self-perception and reality tends to be widest.

Step 3: Practice Deliberate Verification

The single highest-leverage habit for closing the overconfidence gap is building a verification practice. Before accepting any AI output, ask: How would I check this? If you can't answer that question, you've identified a skill gap.

Our guide to critical thinking with AI covers specific techniques — claim decomposition, cross-model verification, and stakes-based review — that build the meta-cognitive awareness Dunning-Kruger research shows is the antidote to overconfidence.

Step 4: Reassess After Targeted Practice

Calibration improves with feedback. After spending time on your weakest dimension, retake the assessment. The goal isn't just a higher score — it's a smaller gap between your prediction and your result. When your prediction accuracy improves, you've developed the self-awareness that separates over-reliance on AI from genuine fluency.

The Calibration Hierarchy: What It Means for Your Organisation

The role-by-role prediction gap data reveals a clear hierarchy of self-awareness about AI skills. Founders and Product managers sit at the top — not because they're the most skilled (though they score well), but because their professional contexts force regular reality checks. Students and Data professionals sit at the bottom — not because they're less intelligent, but because their contexts provide fewer corrective signals.

For engineering managers and HR leaders planning AI upskilling, this hierarchy has direct implications:

1. Don't trust self-reported AI skill levels — the error rate varies by role, making aggregate self-assessment data unreliable for planning
2. Baseline everyone with the same evidence-based measure — AISA provides a consistent, rubric-based evaluation that doesn't depend on self-perception
3. Tailor interventions by role — students need fundamentals and calibration; engineers need to bridge the gap between coding skill and AI fluency; data professionals need practical application beyond their statistical comfort zone
4. Measure again after training — the prediction gap is the metric that tells you whether your programme is building genuine competence or just inflating confidence further

The AI readiness assessment checklist provides a structured framework for implementing this approach across a team or organisation.

Overconfidence about AI skills is universal, but it's not uniform. Knowing how much each role overestimates — and why — is the first step toward building AI programmes that close real gaps instead of reinforcing comfortable illusions.

---

Related reading: AI Skills for Students: 222 Assessed [2026] — the full breakdown of why students score lowest and overestimate most.

Related reading: Over-Reliance on AI at Work [2026 Data] — what happens when overconfidence meets daily workflows.

Related reading: AI Fluency Levels: 5 Tiers Explained [2026] — what each AISA tier actually means in practice.

Frequently Asked Questions

Which job role is most overconfident about AI?

Students are the most overconfident group, overestimating their AI fluency by 31.5 points on average — predicting 66.7 (Proficient) but scoring 35.2 (Developing). This is nearly double the overall average prediction gap of 17.8 points. Founders are the most calibrated, overestimating by only 7.1 points.

Why do students overestimate AI skills?

Students conflate frequent AI usage with genuine competence. They often use tools like ChatGPT extensively for coursework but rarely verify outputs, evaluate model limitations, or consider safety implications. Their lowest-scoring dimensions — Technical Understanding (30.9) and Safety (30.9) — are precisely the areas where overconfidence is hardest to self-detect because the feedback loops are weakest.

How is the prediction gap measured?

Before taking the AISA assessment, candidates submit a self-predicted composite score (0-100). After the conversational assessment, an independent AI evaluator scores their responses against 11 criteria using a published rubric. The prediction gap is the difference between the predicted score and the actual score. Across 1,596 candidates who submitted predictions, the average gap was 17.8 points of overestimation.

Can you reduce your own prediction gap?

Yes. The prediction gap reflects an information deficit, not a fixed trait. Getting an evidence-based baseline, identifying your weakest dimension, and building deliberate verification habits all improve calibration. Founders and Product managers show smaller gaps because their professional contexts provide regular reality checks — you can replicate this effect through structured practice and reassessment.