agentsQ5to verify
Prather et al. — struggling novices finish AI-assisted programming tasks with an 'illusion of competence'
In observational studies of novice programmers using AI coding assistants, struggling novices can complete tasks (with AI scaffolding) while developing a measurable disconnect between visible task performance and underlying code comprehension — the AI substitutes for the cognitive work that would have produced internalized skill, leaving the learner with an inflated sense of competence relative to their independent ability.
Discrepancy between AI-assisted task completion and independent (no-AI) code-comprehension or modification ability among novice programmersQualitative + quantitative observation of completion-without-comprehension; specific effect sizes / N not extracted to verification.
- Sample
- Novice-programmer cohort; exact N not extracted to verification.
- Methodology
- Observational + task-completion study of novices using AI coding assistants, with measurement of independent comprehension separated from AI-assisted task performance.
What this means
- Specific empirical anchor for the 'performance-understanding dissociation' that the AHI longitudinal-cognitive-effects review identifies as the strongest synthesis claim in the literature.
- Implies a measurement gap in current AI-coding evaluations: visible completion metrics systematically over-estimate the underlying skill they are taken as proxies for.
- Direct relevance to Penwright's writing-features evaluation: the parallel claim for writing (visible artifact-completion ≠ writer's internalized capability) is the load-bearing measurement target.
Source
(Title to verify — novice-programmer AI-assistant study showing illusion-of-competence)
Computing-education research (specific venue / paper to verify) · James Prather & et al. · 2024 · peer-reviewed
Context
- What came before
- Computing-education researchers had observed similar performance-comprehension gaps with template-based and search-assisted programming. The AI-assistant case sharpens it because the scaffold is dynamic and conversational rather than static.
- What comes next
- Verify exact study design, N, comprehension instrument. Connect to Qiao et al. (performance up without codebase understanding) and Shihab et al. (brownfield shift to prompt-view-implement) as the related triangle of evidence.
- Where this lands
- Encyclopedia Part I §1.3 (methodology gap — performance/understanding dissociation), Part II (workforce — what AI changes about apprenticeship), Part V (research frontier — what we don't yet know about long-run skill formation).