What AI finds in code that pattern matching cannot

We ran three AI security harnesses, Opus 4.8, Mythos 5 and the Codex security suite, over four production codebases in a commercial multi-tenant product. The corpus is roughly 4.8 million lines, drawn from an estate of more than seventy repositories. The harnesses produced 1,318 findings across 202 distinct CWEs. We triaged the Opus and Mythos exports to reviewer verdicts: 87.3 percent precision for Opus, 91.6 percent for Mythos. Codex was not triaged.

Five results are stable enough to state up front.

The rest of this note is the evidence for those claims, then the limits of that evidence. Figures are observations. What we take them to mean is written after the figure they rest on.

The estate splits into one large managed-code application and three native codebases. That split is the comparison that organizes everything below: the same three harnesses, two language families, and several repeated runs per codebase between June and September 2026.

The 3.8 million line web application produced the largest absolute number of findings and by far the lowest density. Across all three models, the two measured native codebases were roughly six times denser per line.

We compared four populations, not one pooled total. Codebases, because managed and native code do not share a defect distribution. Models, because Opus, Mythos and Codex are different configurations, not three draws from one scanner. Runs, because the same harness over the same code is still a sample. Defect classes, because a finding count is not a coverage statement until it is pinned to a taxonomy.

Opus and Mythos already exported a CWE, so those 694 findings joined a shared twelve-class grouping without a second pass. Codex did not: that export is title, description, severity and paths. To put all three models on one chart we classified every Codex row with Jev 1.13.0, TypeSafe's typed decision model. Each call sent title, severity, language and a scrubbed description, never emails, repositories or file paths. Jev answered three questions in one pass: the twelve defect classes, the CWE catalog Opus and Mythos already used, and whether the chosen CWE was a clear match rather than a forced fit. Class confidence was high, median 0.96. The specific CWE was more spread, median 0.47. Most of the 126 informational Codex rows have no clean CWE, 35 of them untagged; they still sit on the charts so the three models are counted the same way.

What Jev isHow the API worksTypeSafe

A later run counted as the same finding when it named the same file and the same CWE. Precision figures are our own reviewer triage of Opus and Mythos on a single codebase. Codex was never triaged, so no precision figure is quoted for it. Language is inferred from file extensions. Those choices are the scope of the comparisons below; the limits they impose are in the Limitations section.

Across all runs, 492 of 1,318 findings fall into authorization, authentication, object-level access and race classes: 297 authorization and RBAC, 110 authentication and session, 45 IDOR and 40 races. On the multi-tenant web application that share is 62 percent.

The top three CWEs account for 259 findings between them. None of them has a dangerous function at the center. A taint rule works by recognizing untrusted data arriving at a sink. There is no sink here. The defect is that a check which should exist does not, or runs in the wrong order, or validates the wrong subject.

CWE-639 is the clearest case. An endpoint accepts an identifier, looks up the record and returns it, and every individual step is correct. The bug is that nothing ties that identifier to the caller's tenant. To see it you have to know that the application is multi-tenant, know which table carries the tenancy column, and know that this particular handler is reachable by an authenticated user of a different tenant. That is application knowledge, not a syntactic pattern. A linter rule for it would either never fire or fire on every database read in the codebase.

Role-based access control fails the same way. A role is checked, but after the data is fetched, or against a role that is necessary rather than sufficient, or on the wrong object in a delegated session. Race conditions are harder still: the 22 time-of-check time-of-use findings and the 40 race findings overall depend on two operations being separated in time under concurrency, which is a property of execution rather than of syntax. Path-sensitive analyzers can model some of these; in practice they accumulate false positives long before they reach the interesting ones.

The 1,318 findings carry 328 distinct category labels, and 180 of those labels appear exactly once. That long tail is where business logic lives: a missing state-transition validation, a fail-open authorization branch, an SSO state fixation, a path equivalence bypass, a capture-replay authentication bypass, an encoding confusion between two layers that each behave correctly alone. A rule for a defect that appears once, in one application, has no second customer. A model that reads the code can still surface it.

We went through the public Semgrep rule set. Pattern matching is built for secrets and injection. The three CWEs that dominate this estate barely appear, and none of those files is a Kotlin or Java tenant check. The pack also has no feedback loop that would turn a confirmed finding into more variants for the next AI pass.

Application context also decides which of these classes exist at all. Sorting every finding by the language of its file makes that split measurable.

Memory safety lands 113 findings in C and C++ and exactly zero in Kotlin or Java, which is the one result here that a runtime guarantees rather than a scanner. Filesystem is 48 native against one in Java. Object-level authorization stays a managed-code problem: 26 of 33 IDOR findings are in Kotlin or Java, because that is where the multi-tenant request handlers live. Authorization overall is the class that shows up everywhere. Codex is what thickens that band: 83 of its 403 findings, 46 of them in Kotlin on the web application.

What we take this to mean: the defect classes worth hunting are a property of the codebase, not a global setting. A native agent needs memory safety and filesystem race coverage. A multi-tenant application server needs authorization, tenant isolation and object-level access coverage, and gains nothing from a memory safety pass. One scanner configuration for both produces a queue that can be technically accurate and still aimed at the wrong classes.

Mythos found about 2.6 times the raw volume of Opus. Codex sits between them at 403 findings, 277 of them actionable. Volume alone would pick Mythos. Normalizing each model's findings against its own total shows a different picture: the three configurations are good at different things, and the gaps are large enough to measure.

Opus devoted a fifth of its output to memory safety, against 9.6 percent for Mythos and 6.5 percent for Codex, and it leads on crypto and transport defects. It also grades hotter: 38 percent of its findings rated high, against 28 percent for Mythos and 5 percent for Codex. Mythos owns the concurrency and hygiene classes: it found races at 6.6 times the Opus rate, secrets and exposure at 3.9 times, and filesystem defects at 2.7 times. Codex spends 21 percent of its output on authorization and 17 percent on integrity, and almost nothing on injection. Jev put 16 of Codex's 20 highs on CWE-862, and every one of those 20 highs arrived with a proposed patch. File-level overlap with the other two harnesses was 0 to 3 percent, and zero on the web application.

The same split shows up by language. Opus put 69 percent of its findings in C and C++ and 7 percent in Java. Mythos spread wider, at 53 percent native, 12 percent Java and 5 percent shell scripts. Codex is 57 percent native and 22 percent Kotlin, which is where its authorization budget went. The per-CWE view is consistent: Opus over-indexes on out-of-bounds reads and writes, server-side request forgery and path traversal; Mythos over-indexes on time-of-check races, link resolution, file permissions and credentials in logs; Codex over-indexes on missing authorization.

What we take this to mean: the useful question is not which model is best. It is which model is best at the class you decided to hunt, on the language you are hunting it in. One frontier model on one pass leaves whole categories uncovered. A report does not list what the model was bad at, so those gaps are not visible to the buyer of a single opaque pass.

Matching by file and CWE, repeated runs over the same code almost never agreed on the individual finding.

Five runs over the endpoint agent produced 382 distinct findings. The best single run found 116. On the native service, three runs against an identical commit agreed on 0, 0 and 1.1 percent of findings. Finding-level agreement elsewhere sat near 2 percent.

The instability is at the level of the individual finding, not the class. On the web application, authorization ranked first in every run at 43, 60 and 56 percent of findings. Roughly half of any run's findings fell inside another run's eight most common CWEs. Five pooled runs over the endpoint agent returned 3.3 times the distinct findings of the best single run.

The labels moved too. Those 1,318 findings carry 328 distinct free-text category labels. A single concept, missing or incorrect authorization, appeared under fifteen different labels, including missing authorization, improper authorization, incorrect authorization, authorization bypass, fail-open authorization, IDOR, IDOR/BOLA, broken object level authorization, and three separate spellings of that last one. Group by CWE and the same data is stable: 120 findings at CWE-862, 76 at CWE-639, 63 at CWE-863.

What we take this to mean: you cannot call a release clean on a single draw, because the next run returns a different set. You cannot tell whether last quarter's remediation reduced risk, because a drop in findings is indistinguishable from a different sample. You cannot gate a pipeline on a fresh draw, and you cannot dismiss a finding with confidence because the corroborating run never happened. The class distribution is stable enough to aim at, budget against and measure. The individual finding is not. If runs barely overlap they are not redundant, so pooling compounds rather than repeats, but only after every finding is normalized onto a fixed taxonomy. A dashboard built on model-authored labels is not a measurement.

At the point we pulled this data, all 1,318 findings were still open and none had been triaged to a verdict. The measured median reviewer time was 99 minutes per finding, which puts about 1.36 engineer-years between this report and any change to the product. In one export, 20 of 403 findings came with a proposed patch, about 5 percent. Whole-repository passes run for hours. Roughly twenty repositories in this estate could not be onboarded at all because the scan exceeded the available budget.

Each finding costs 99 minutes because a model assertion is not a decision. The reviewer relocates the affected path, works out whether anything upstream already enforces the missing check, decides whether the path is reachable from a real entry point, judges the blast radius against the tenancy model, then designs a fix that does not break a caller. The report contributed a claim and a severity label. The reviewer rebuilds the analysis that justifies acting on it.

What we take this to mean: detection was never the constraint. The scarce work is deciding what deserves action and attaching evidence a reviewer does not have to recreate. Hours-long whole-repository passes also keep that work outside the pull request, so everything above is archaeology, found after the code shipped. Scoping to the changed lines and the code reachable from them is what moves the same machinery into a merge gate. It costs a few dollars for a 200 line diff, against well under one percent of a full-corpus pass for a 2,000 line one.

These figures come from harness runs over one commercial multi-tenant estate, anonymized, between June and September 2026. They are not an industry benchmark, and we would expect the magnitudes to move on different architectures.

AI reads code well enough to find missing authorization, tenant isolation failures, object-level access bugs, RBAC ordering mistakes and time-of-check races that no rule-based tool in this estate had reported. That much is clear from this corpus. What this corpus does not show is how to make that detection operable: reproducible enough to manage, attributed to the right model for the class, normalized enough to measure, substantiated enough to act on, and fast enough to gate a merge. Those are properties of the analysis system, not of the model.

What we will measure next: whether class-level stability holds on other architectures, whether a validator whose job is to disprove a finding compresses the 99 minute review, and whether scoped passes can move this machinery into a merge gate without inheriting the recall gaps of a single opaque run. Cost, deployment and the shape of that system belong in a later note.