2023 United States meta

Do Users Write More Insecure Code with AI Assistants?

Neil Perry, Megha Srivastava, Deepak Kumar, Dan Boneh — Stanford, CCS 2023 — yes, and they believe the opposite. The illusion is the danger.

Neil Perry, Megha Srivastava, Deepak Kumar, and Dan Boneh, at Stanford, ran the first large-scale controlled study of security outcomes when programmers use AI coding assistants. Forty-seven participants completed five security-relevant programming tasks across Python, C, and JavaScript, with random assignment between AI-assistant access (OpenAI codex-davinci-002) and a control group. Result: participants with AI access wrote significantly less secure code across the majority of tasks. Worse, they were also more likely to believe their code was secure than the control group. Participants who engaged more critically with their prompts — re-phrasing, adjusting, questioning the suggestions — produced safer code than those who accepted suggestions verbatim. Published at CCS '23, the flagship academic security conference. The first peer-reviewed evidence that AI coding tools degrade the global property humans most need to preserve under change: security.

In plain terms

The Stanford team set up a controlled study. Forty-seven programmers. Five security-relevant tasks across three languages. Half the participants got an AI assistant — Codex, the model that powered the original GitHub Copilot. The other half got nothing but their editor. All other variables held constant. The result was not subtle. Across the majority of tasks, the AI-assisted group wrote less secure code. More buffer overflows. More SQL injections. More authentication bypasses. More misuse of cryptographic libraries. This was measurable, statistically significant, and consistent across languages. The second result was worse. When asked, the AI-assisted group also reported higher confidence that their code was secure. They were producing more vulnerable code while feeling better about it. This is the inversion that should have ended the conversation about AI as a security tool — the worst possible combination is "your code is more vulnerable AND you trust it more." The third result is the one that points the way forward. Within the AI-assisted group, participants varied in how they used the AI. Some accepted suggestions wholesale. Some engaged critically — re-phrasing prompts, comparing alternatives, questioning the model's claims. The critical-engagement subset produced safer code than the wholesale-acceptance subset, and roughly as safe as the control group. The lesson is sharper than "AI is bad at security." The lesson is about the epistemics of code authorship. When you write code yourself, you have a calibrated sense of what you understand and what you do not. When you accept code from an AI, you import the AI's confidence along with the code — and the AI is confident about everything, because the model has no calibrated uncertainty. The user fills in the missing calibration with their own confidence in the model. The code becomes harder to review because the author is not present to be questioned. This is the empirical leading edge of the book's thesis. Software maintenance is the work of preserving global invariants under change. Security is one of those invariants — perhaps the load-bearing one. AI tools push local productivity up and global invariant preservation down, and push the user's confidence in the result up. The Stanford study is the first peer-reviewed measurement of this trade and the first to quantify the confidence-vulnerability inversion. The illusion is the danger. Confidence inversely proportional to safety.