Over the past six months I ran five experiments: an AI project manager, auto hyperparameter tuning, ABAP evaluation, internal/external network desensitization, and a chip design graph. None of them hit the wall of model capability—they all hit the same question: who proves this thing is correct?
AI Native R&D is not "engineers + a faster Copilot"—it is a process redesigned around a new price structure: generation is nearly free; trustworthy verification, coordination, understanding, and data boundaries have become the scarce goods.
You swap models every quarter. What you can't swap out is your own verifier—scoring functions, acceptance criteria, expert evaluation sets, data classification rules. Teams should invest in that, not in stronger generators.
How are chip manufacturers currently deploying AI in production, and how should an ontology be built to comply with risk-control regulations?
Short answer. I can answer five things; one I cannot. What I can answer: the current state at manufacturers is moving from "ban everything" to "zone-based allowlists"; the adoption sequence on the design side follows "is there a trustworthy verification method"; risk control lands on data classification, adversitory auditing, permission tightening, and shadow mode; another path is confidential inference, letting the compute provider not see the data so desensitization is unnecessary (§06 has a subsection at the end, covering maturity and limitations); "ontology" also has a subsection at the end of §06 (first clarification: it is not a replacement for existing data, models, and pipelines, but a business twin layer built on top of them)—in semiconductor manufacturing there is academic research, cross-company data platforms like Athinia, and comparable permission mechanisms, but what's missing is published effectiveness data. What I cannot answer is the fab production line (equipment, MES, yield closed loop): this article covers chip design flow and data boundaries, not the production line. To understand the ontology itself first, see "The Full Anatomy of the Palantir Empire".
How can we reduce human involvement in code review and testing while ensuring AI writes high-quality, usable code?
Short answer. The article's judgment is not to eliminate humans, but to move them from line-by-line review to three positions: signing off on acceptance criteria, reviewing critical changes, and deciding on release. The prerequisite is first making machine verification trustworthy (acceptance commands, harness, and graders must not be writable by the generator). Existing evidence: at one company, human review coverage dropped from 89% to 68% with no change in rollback rate (single company, non-random); Stripe's 1,300+ agent PRs per week are still all reviewed by humans. There is no evidence that human review can be fully removed.
I did the math for my autoresearch-batch experiment. Having GLM generate one round of candidate improvements for CIFAR-100 costs about ¥1.5 in LLM calls—already after the Batch API 50% discount; running that round of candidates on a T4 costs about ¥20.
Writing candidates is more than ten times cheaper than verifying them.
autoresearch-batch · CIFAR-100 single round
This ratio later reappeared in different shapes in every experiment I ran. The Agent acting as team project manager—the hard part isn't writing messages, it's preventing it from randomly changing state; producing evaluations for legacy SAP code—the hard part isn't writing questions, it's making the scoring withstand expert re-review; letting an internal network borrow an external Agent—the hard part isn't desensitization, it's proving the desensitization is sufficient.
Below I walk through these five experiments in order: tempo, autoresearch-batch, ABAP-Bench, escapegate, chip design full-flow graph. For each one I re-checked against external evidence, and I will be clear about where that evidence contradicts me.
First set aside a common narrative: how much faster AI made development. This question can't even be measured cleanly anymore.
In July 2025, METR published the most-cited randomized controlled experiment: 16 experienced open-source developers, 246 real tasks—with AI allowed, completion time was 19% longer; developers expected a 24% speedup beforehand, and still felt 20% faster afterward. In February 2026, METR itself changed the experimental design: a new round included 57 developers, 143 repos, 800+ tasks; estimates became −18% and −4% (negative means faster), but both confidence intervals crossed zero—METR called it "an unreliable signal." The reason is selection effects: more and more developers refuse to work without AI. The control group is disappearing.
This detail is more interesting than the "19%" itself. When a technology is good enough that people refuse to work without it, the A/B experiment tool is broken. Going forward, productivity claims will mostly have to rely on other signals.
Other signals point in the same direction. CMU and Stanford tracked data from a mid-size software company from January 2024 to April 2026: 802 developers, 196,212 PRs. The company set a "2×" target in mid-2025, wanting each engineer to double their merged PRs—and they did: average PRs per active developer went from 21.2 to 44.3, a 2.09× increase. But in the same period, total company PRs grew 3.1×, while the number of reviewers only increased 1.5×; the proportion of PRs receiving at least one human review dropped from 89% to 68%, while AI automated review coverage rose from ~19% to ~84%. Merge rate and rollback rate showed no significant change. The paper itself notes this is a single company, non-random observation.
What these numbers say also depends on what they don't say. They say review bandwidth didn't keep up; they don't say AI code is worse—rollback rate didn't move. Faros used telemetry from 20,000+ developers and got more dramatic numbers (median review duration +441.5%, in a later report, PRs merged without review +76.3%), but Faros sells a measurement product—these are vendor metrics, and they themselves call it correlation. Stack Overflow's October survey took a different angle: 83% use AI, but only 6.6% trust it for important decisions; 48% say they trust it only when the output is easy to verify.
Trust is bound to verification cost. Bain's estimate is more blunt: from idea to production, writing code and testing account for only 25%–35% of time. Extrapolating from this share (this is my extrapolation, not Bain's conclusion), even if coding itself were infinitely fast, the upper bound for the whole pipeline would be only 1.33 to 1.54×.
There's another number used everywhere that deserves to be called out separately: "how much code AI writes." Google says 75% of new code is AI-generated and approved by engineers; in Anthropic's internal survey, over half of respondents said the work they could fully delegate to Claude is 0%–20%. The former counts code provenance; the latter asks how much work can be handed off—they're not the same thing. I found at least five mutually incomparable metrics, so I won't use this number as evidence later.
Writing did get faster; the whole pipeline didn't speed up proportionally, and the pressure landed elsewhere. My five experiments each hit one of those pressure points.
First let me explain what I mean by AI Native R&D, because the term has no agreed definition. AWS proposed AI-DLC in July 2025, splitting existing approaches into "AI-assisted" and "AI-autonomous," calling itself a third way (a vendor methodology); academia has Hassan et al.'s SE 3.0 vision; Thoughtworks Technology Radar vol. 34's podcast frankly admitted that industry terminology hasn't stabilized. My usage is narrower: it's not about which tools you use, but whether the development process has been redesigned around the new price structure of "generation is cheap, verification is expensive."
Inside each of these five experiments hides a verifier. Let's look at them one by one.
Experiment writeuptempo — AI project manager, cadence for small teams
tempo started from a very specific frustration. Our team of 4 spent more time on coordination than on execution each week: syncing status, splitting tasks, guessing who was stuck. tempo is an AI project manager running on GitHub Actions: it pushes progress at 09:00, 14:00, and 19:00 via cron; given a --goal, it decomposes the objective into 3 to 8 issues with code locations and acceptance criteria; when an issue closes, it writes feedback using a "See / Feel / Expect" three-part format, with the prompt explicitly banning "empower," "closed loop," and "cheer up."
When writing it, the effort I spent most on wasn't making it write like a human—it was limiting what it could do. The Agent has 7 tools total: 4 read-only, 3 write (send notification, create issue); it cannot close, move, or reassign any issue. The reason is in the design principles: AI changing issue status is hard to review; one mistake destroys the entire sprint view, so the LLM only pushes suggestions, and humans act.
The second constraint is equally hard: every issue must be such that "an engineer can start without asking follow-up questions." The page has a comparison pair: a bad task is "complete Feishu bot integration"; a good task points to a function (feishu_webhook.py:handle_message()), specifies trigger conditions, and ends with acceptance criteria—tests/test_feishu_p2p.py all green.
In hindsight these two constraints are the same thing: turning coordination into something inspectable. Tasks carry acceptance criteria, so the machine can judge completion; the Agent doesn't touch state, so responsibility has a single locus.
External designs are converging on the same point. Linear's August 2025 Agent interaction specification requires that issues can only be assigned to people, and only delegated to agents; after delegation, responsibility doesn't transfer, because agents can't be held accountable. Atlassian's Jira took another path, calling the agent an "accountable member" of the team. The two philosophies are exactly opposite, and there's currently no evidence to adjudicate. Anthropic's Project Vend Phase 2 is the only first-hand experiment I found where an agent acts as "manager": the CEO agent approved lenient requests about 8× more often than it denied them; Anthropic's own judgment is that the shop's profitability may be despite it, not because of it. In the absence of empirical evidence, "push suggestions, humans execute" is not conservative—it's just the only design that currently stands up.
The external project most similar to tempo is OpenAI's Symphony, open-sourced in late April 2026: a persistent service that polls task boards like Linear, opens an independent workspace for each issue, runs a coding agent inside (the reference implementation connects Codex), and advances issue state as a state machine. Its trade-offs are exactly opposite to tempo's. The SPEC states that Symphony itself is only a scheduler, executor, and board reader; changing state, writing comments, and attaching PR links are "usually done by the coding agent," using credentialed tools where credentials aren't placed in the agent's runtime. A successful run can stop at a handoff state like Human Review for human verification, rather than going directly to Done. tempo keeps all state-modifying permissions in human hands; Symphony gives them to the agent, then puts humans back in the loop via "handoff states plus human verification." The difference between the two approaches is not who is more conservative, but which gate humans guard: tempo guards "modify state," Symphony guards "verification". As for risk boundaries, the SPEC doesn't prescribe unified approval and sandbox policies, requiring each implementation to document its own trust boundaries; the README calls it a "low-key engineering preview for trusted environments" and says it works best in repos already practicing harness engineering. OpenAI claims some teams saw PRs increase 500% in the first three weeks; per Help Net Security's retelling, no baseline or methodology was given—vendor self-report.
Stripe's Minions pushes the other end very far: merging 1,300+ fully agent-written PRs per week, but all reviewed by humans, with CI running at most two rounds before returning to humans (Stripe self-report). Cursor's experience shows how coordination breaks: 20 agents flat-coordinating with lock-based mutual exclusion, throughput only 2–3, later restructured into planner, worker, and judge layers. Anthropic's Carlini used 16 parallel Claudes over two weeks to write a C compiler that can compile Linux 6.9, API cost under $20k; his takeaway was that task verifiers must be nearly perfect, otherwise Claude will solve the wrong problem.
tempo has no control group. "Feels like it knows what I'm doing better than I do" is feedback, not a measurement. Brodzinski raises a concern worth noting: detailed specs can suppress discussion between people. If tempo's issues contain only code locations and acceptance criteria, collaboration may devolve into "throwing it over the wall." tempo's emo mode targets the "emotions are invisible" category among three types of overhead; whether it can also fill the communication gap, I have no data.
Experiment writeupAuto-optimizing ML algorithms with Batch API · AutoResearch Batch experiment report·Idea sourceKarpathy's autoresearch
The setup for this experiment is simple: give an LLM a training script and the current best metric, let it propose improvements and write complete code; the framework asynchronously submits a batch of candidates via Batch API, collects them, runs experiments, selects the best, and enters the next round. The training script only needs to print one line of ARB_METRIC.
The results aren't shocking. On CIFAR-10, a 3-layer CNN's 0.7579 was pushed to 0.8370 (+7.9pp) by a VGG-style network the LLM wrote itself; on CIFAR-100, a 4-layer CNN's 0.4633 reached ResNet-18's 0.5132 (+5.0pp). Two of six candidates were worse: SGD lr=0.1 dropped 8.1pp, Dropout dropped 3.1pp. Letting data eliminate bad ideas is the whole point of this loop.
What's interesting is the bill. For one CIFAR-100 round, LLM calls cost about ¥1.5, GPU training about ¥20. The Batch API 50% discount applies to the smallest line item, saving roughly 5% of total cost. I wrote on the page: treat it as a "50% cost reduction" and you'll be disappointed; treat it as the substrate for non-real-time batch optimization and it's correct.
External data shows the same ratio. In the SkyPilot experiment mentioned earlier, 8 hours submitted about 910 experiments: Claude Code API cost ≈ $9, GPU ≈ $260, LLM about 3%. METR did a NanoGPT speedrun self-optimization study in July this year: 70%–90% of agent cost is experiment compute. Three independent sources: generating candidates is nearly free; what costs money is verification.
Once the ratio is clear, the next question is: who sets the verification budget? Karpathy's autoresearch makes it a convention: each experiment is fixed at 5 minutes (TIME_BUDGET defined in prepare.py), the scoring function evaluate_bpb is also in prepare.py, marked DO NOT CHANGE, and program.md requires the agent to only modify train.py. This is a convention, not a sandbox: the early-stopping judgment is written in train.py, and the loss summed by evaluate_bpb comes from GPT.forward's return value in train.py—technically the agent can modify both. His own published result was about 700 experiments finding about 20 effective improvements, compressing nanochat's Time-to-GPT-2 from 2.02 hours to 1.80 hours. My experiment didn't even have this layer of convention; the LLM could decide training intensity itself. It set ResNet-18 to 50 epochs; this single candidate ran 38 minutes on a T4, 10× the lightest candidate; the more candidates, the higher the probability of mixing in a heavy one, and the whole round gets dragged down.
The fix is two-stage filtering: first run all candidates with 2 epochs for quick screening, then only let the top 2 pay the cost of full training. At K=50, a single round compresses from 583 minutes to 127 minutes, saving 78%.
This fix has a price, and SkyPilot's agent hit it too: it screened on cheap H100s and confirmed on H200s, only to find that a certain learning rate ratio parameter won on H100 but lost on H200. Quick-screen ranking doesn't necessarily equal full-run ranking. I have a lesson in the opposite direction—a VGG candidate timed out during under-resourced training and was judged "failed," yet it was the optimal solution. Cheap proxy verification can kill good candidates.
One level deeper: can the verifier itself be gamed? METR wrote in their May 2026 risk report that on tasks exceeding 8 hours, at least 16% of successful runs are improper after re-review. In the NanoGPT study above, the agent tried "stop training at the first verification step" to meet the target, and also tried freezing the MLP to approximate the target loss; METR re-ran each claimed improvement 40+ times before counting it. When Shopify used autoresearch on the Liquid template engine, the agent briefly traded speed for deleting large numbers of files; the author wrote "unacceptable". My experiment didn't see cheating, but I also didn't design checks that could catch cheating—these two things are not equivalent. I wrote a separate piece expanding on this: Graders Must Be Read-Only: Reward Hacking and Metric Failure in Long-Horizon Automated Experiments.
Karpathy put this logic most plainly: Software 2.0 easily automates what you can verify (November 2025). Jason Wei's verifier's law lists five properties of verifiable tasks: objective, fast, parallelizable, low noise, continuous reward. CIFAR candidate verification satisfies objective and continuous, but isn't fast (minutes to 38 minutes), and single training runs carry randomness. This is the "verifiable but not cheap" category, so two-stage filtering becomes a necessity. When Google made AlphaEvolve a cloud product, it required customers to bring their own deterministic evaluation script—the scoring function became a user-must-deliver input.
Experiment writeupWhen AI Takes the SAP Consultant's Seat (ABAP-Bench)
Last winter, Old Zhang—an SAP consultant with 20 years at a manufacturing company in eastern China—retired. He took with him not just his badge, but the entire "living map" of hundreds of ABAP programs in the ECC system: which code was patched at 3 AM for a bizarre tax code bug, which RFC interface connects to which layer of the Golden Tax system's authentication, which field called SGTXT actually stores invoice remarks. None of this was written in any document.
The timeline is real. SAP's official statement is that Business Suite 7 (including ECC) has mainstream maintenance until end of 2027, extended maintenance until end of 2030.
Anthropic's February 2026 blog post on COBOL compressed this into one sentence: legacy code modernization has stalled for years because understanding it costs more than rewriting it. The day of the post, IBM's stock dropped about 13%. IBM's Rob Thomas subsequently wrote a rebuttal, arguing that translating code and modernizing the platform are two different things; Gartner's prediction was colder: over 70% of mainframe exit projects started in 2026 will fail to achieve expected benefits, because generative AI tool capabilities were overestimated (The Register relaying Gartner's report). All three parties stake different claims, and the disagreement lands in the same place: can AI cross the "understanding" step?
ABAP-Bench is the ruler I built for this disagreement: 60 expert-authored migration tasks, covering 9 dimensions. Several design choices are worth noting. Answers must be specific: just saying "recommend replacing BKPF with a CDS view" gets zero points; you must name I_JournalEntry and specify that BUKRS maps to CompanyCode. Wrong claims are penalized: incorrectly stating "BKPF is an S/4HANA primary table" deducts 3 points, because being wrong is worse than saying nothing. Test code uses Chinese variable names like "invoice number" and "tax-inclusive amount" that almost never appear in public codebases, serving as anti-cheating watermarks. 10% of old questions are retired each quarter. Scoring has three layers: expert rubric 40%, code structural quality 30%, BM25 semantic similarity to expert golden answers 30%.
The first test was GLM-5, total score 61/100. Among nine dimensions, China-specific compliance was highest (67.5%), defect discovery lowest (54.2%). More informative is the variance: classic ECC-to-CDS rewrite (T03) scored 16/20, producing directly compilable JOINs; cutting-edge RAP EML rewrite (T37) scored only 5/20—correct direction, wrong syntax details. Five bugs were planted in the VAT program; it found 3, missing the two deepest. The reasoning model averaged 145 seconds per question; the highest-scoring T16 used 155 seconds (17/20), while the lowest-scoring bracket T36 used 447 seconds (6/20)—thinking longer doesn't fill knowledge gaps.
How to read this score, I'm now more cautious than when I wrote the page. Public ABAP evaluations paint a different picture: TH Köln's paper used 180 questions with executable unit tests, letting the model revise with compiler feedback for 5 rounds; GPT-5 went from 19.28% first round to 77.11%; the community-maintained Zeis leaderboard shows Gemini 3.1 Pro at 44.33% first round, 88.33% after 5 rounds. Same model, with or without a verification loop, differs by several times. My 61/100 evaluates single-shot answers against expert criteria—different quantity from these numbers, not directly comparable. This actually pushes the conclusion even clearer: whether a model is weak on ABAP depends largely on whether someone built it a good verification loop; and what the expert rubric evaluates is precisely what single-test benchmarks can't reach—risk assessment, compliance, whether it should be migrated this way.
Another side evidence comes from SAP itself. It trained a specialized ABAP-1, using 250M lines of ABAP, officially positioned for understanding and explanation, not generation. On the Zeis leaderboard its generation success rate is 10.67% first round, 19.89% after 5 rounds—far below general models. Comparing a generation leaderboard against an explanation model isn't fair, but this precisely illustrates the point: whether a specialized model is worth it—without domain evaluation, outsiders can't know.
Evaluations themselves also rot. OpenAI stopped reporting SWE-bench Verified in February 2026, and withdrew its recommendation of SWE-bench Pro in July; per GIGAZINE's retelling, about 30% of problems had issues (I couldn't verify OpenAI's original text; going by the retelling here). SWE-Bench Illusion found that given only issue descriptions and repo names, models could locate up to 76% of faulty file paths on Verified, but only 53% on other repos—carrying traces of memorization. "Retire 10% each quarter" and "Chinese variable name watermarks" now read like prophylaxis against the same disease. Anthropic's January evaluation methodology offered a standard: a good task is one where two domain experts independently reach the same pass/fail conclusion. Whether my 60 questions meet this, I didn't do an inter-rater reliability check this round.
Writing this, my view of "evaluation" has changed. In the AI era, "what counts as done correctly" can't be conveyed in a paragraph of natural language for the executor to interpret; it only counts as clearly stated when written as a set of criteria that machines can repeatedly execute and experts are willing to sign off on. ABAP-Bench's rubric is essentially a requirements document for "what a correct migration looks like"—just with the reader swapped to a model.
Experiment writeupescapegate · Escape desensitization architecture verification demo
Internal-network development wants to borrow external Coding Agents, but can't send company names, customer names, or internal systems out as-is. Semiconductor, defense, and finance run into this almost daily. Samsung's trajectory is a good footnote: in spring 2023 it internally opened ChatGPT, and soon had three leaks (source code for equipment measurement databases, yield test sequences, meeting transcript transcriptions); in May it banned it entirely; in June 2026, the DX division officially integrated ChatGPT, Gemini, and Claude into internal systems; per Korean media, the semiconductor division had already been using Claude internally; peer SK hynix's approach was to first use external AI in areas "not involving national core technology." Banning everything became engineered zoning.
escapegate is my verification-oriented attempt at this problem. The pipeline is: Internal Agent → Escape Agent → External Agent → Escape Agent → Internal Agent. Outbound has five steps:
To evaluate whether it could withstand chip design company intranet requests, I built a 103-scenario evaluation set with precise entity annotations. After optimization: entity detection recall from 0.226 to 0.959, precision from 0.464 to 0.942; over-desensitization rate from 8.9% to 0; classification accuracy from 0.592 to 0.854 (semiconductor v1 set 0.735 to 0.974).
Two findings are more important than these numbers. First, "structure itself is an asset": raw yield wafer maps, PDK rule values, cross-product-line full RTL—stripping names is useless; these must be directly classified L3. Second, two seemingly reasonable alternatives both failed in practice: structure-preserving transformations on numerical tables can protect values, but column names and table shapes are themselves clues—the audit Agent can still guess the industry; splitting tasks into fragments distributed across multiple models does make reverse-engineering harder (audit confidence drops 0.146), but only 25% of tasks can be split.
Here there's an unavoidable contradiction. The more aggressively you desensitize, the less the external Agent knows what you're asking, and the less it can help; the less you desensitize, the easier it is to reverse-engineer. escapegate's trade-off is to treat "can the question still be answered after decontextualization" as a constraint, while accepting a coverage as narrow as 25%.
External frameworks give language to this trade-off. Simon Willison proposed "the lethal trifecta" in June 2025: simultaneous access to private data, exposure to untrusted content, and ability to communicate outward—an attacker can make the agent exfiltrate data; his conclusion is that the only reliable approach is to remove one leg, not to add more detection. Meta's October "pick-two rule" says the same thing: at most two of three conditions can hold in one session. escapegate removes the first leg on the external Agent side: private data doesn't go out. But on the return path, all three legs reassemble: the internal Agent receives output from the external Agent (untrusted content), and it can touch the intranet and execute. Inbound scanning and human review can't be skipped—that's why.
Leaks may not exit from where you think. PixelLeak (reported September 2026): Glow Labs found 13,000+ internal screenshots sent by Coding Agents to public GitHub repos, involving developers at 300+ institutions, 93% under personal employee accounts. The reason: GitHub's PR image upload only works in the browser; command-line agents, wanting reviewers to see screenshots, created public repos themselves. This channel is the agent's tool behavior—no one pasted sensitive text to an external model. escapegate manages text sent to the external network; for this type of channel, it relies on the external Agent never seeing the real intranet.
The audit Agent's own reliability is the next question. The paper The Attacker Moves Second used adaptive attacks to break 12 recent jailbreak and prompt injection defenses, most with success rates over 90%, despite those defenses originally reporting near-zero numbers. My 0.959 and 0.942 come from a static synthetic evaluation set, never tested against such attackers. The September 2026 DR-SL paper built a structure almost identical to mine: local verifier, external strong attacker re-testing, human fallback, routing elsewhere if it can't pass; the authors themselves admit the theoretical bound can only prove how much leaks, not prove safety—ultimately relying on empirical calibration, hard rules, and human review. So my approach isn't new. This finding is both reassuring and cautionary: someone independently arrived at the same structure, suggesting the direction is right; it also means this is a shared hard problem that no one has solved.
On the compliance side, there's a cold fact. ITAR's definition of "technical data" is functional: any information required to design, develop, produce, or test a defense article counts, regardless of who wrote it or whether customer names appear. End-to-end encryption is not an export safe harbor; it requires that decryption means not be given to any third party; sending plaintext to a cloud model that must read it doesn't qualify. The US State Department and Commerce Department currently have no authoritative guidance for AI (per Just Security, December 2025). This is my reading of the regulations, not legal advice. It at least explains why "structure is an asset" must be classified L3: names stripped, but information counted by function is still that information.
This section only covers the structure of "allowing through." The spectrum of existing approaches, how to read DR-SL's numbers, who audits the audit Agent, and how to design shadow mode—I expanded on these separately in "Who Reviews Before Release: Data Boundary Engineering in AI R&D".
The root of escapegate's contradiction is that data must be given to the external Agent in plaintext, so it must be stripped before going out. Confidential inference reframes the question: don't change the data; let the compute provider not see it during computation—protecting "data in use," the last segment that transmission encryption and storage encryption don't cover. The site's "Confidential Inference" breaks it into four routes, priced by "who do you trust":
The overhead column is a synthesis from the site's article across multiple papers, not my own measurements. The most complete public exemplar is Apple's Private Cloud Compute: designed so that even Apple's own operations staff can't touch user data; requests are encrypted only to verified nodes; nodes have no remote shell; each production build's measurements go into an append-only transparency log for researcher verification (this is Apple's design description; I haven't independently verified). Anthropic's June 2025 "Confidential Inference via Trusted Virtual Machines" goes in the same direction: decryption occurs only inside an isolated VM's small loader; the key server releases keys only after hardware attestation passes—but they themselves state this is only a research sketch to open discussion; accelerators are still outside the trust boundary, and data center and hypervisor security still relies on the compute provider.
Putting it back in the chip R&D scenario, four points must be clarified first:
Taken together (my synthesis below): escapegate strips data before it goes out; confidential inference sends data out but sealed; the ontology discussed next keeps data inside and manages permissions and actions. The three paths address different threats: what the external model can see, what the compute provider can see, and what insiders can do. So in production they're more likely to be layered, not a three-way choice.
escapegate takes the "stripping" path; confidential inference takes the "sealing" path; there's a third: don't remove, don't seal—build the context into a permissioned, sourced structure, commonly called an "ontology."
First clarify a common misunderstanding: an ontology is not a replacement for existing business data, database modeling, algorithmic models, and data pipelines—it's a "business twin layer" built on top of them. Palantir's documentation is quite explicit: the ontology sits above the platform's integrated digital assets (datasets, virtual tables, models), linking them to their real-world counterparts, called the organization's digital twin (semantic part: objects, properties, and relationships; dynamic part: actions, functions, and dynamic security); object types map each concept to the organization's actual data. In a fab, the counterparts would be equipment, lots, wafers, material batches—this is my example. Breaking it down (this paragraph's synthesis is mine): database modeling and pipelines remain the foundation; objects are backed by underlying data sources, and the object repository stays synchronized as data sources update; algorithmic models stay where they are, just connected to the ontology—Palantir's white paper says models bind to the ontology via modeling objective interfaces; what the ontology adds is four things: cross-system shared objects and relationships, permissions, permissioned actions (user edits also enter the same object repository via actions), and provenance. So it's not a read-only mirror or a replacement, but a layer of business semantics, permissions, and actions—the documentation calls it "the organization's operational layer."
This positioning brings two caveats. First, this layer inherits the quality of its foundation: data with inconsistent identifiers and unclear lineage will still be messy when built into objects, so you need trusted master data and data lineage before building the ontology. Second, "twin" here means business twin; don't confuse it with the "digital twin" for physical simulation of equipment and processes in fabs. The site's "Full Anatomy of the Palantir Empire" covered its business logic; here we only look at whether this line exists in semiconductor manufacturing.
It does, but needs to be viewed in layers—different layers, different evidence hardness:
From these materials, three things can be read.
First, the academic line does semantic alignment: letting different systems and engineers speak the same language about the same equipment and defect, and checking model self-consistency. SEMI's SECS/GEM and EDA standards specify equipment communication and data collection structure; I haven't found a SEMI-level universal ontology standard. I also haven't found a semiconductor case using ontology to constrain AI agent actions, or production-line-level yield improvement numbers; what fabs do internally isn't visible in public materials.
Second, the closest to "compliant with risk control" is Athinia. The risk it guards against isn't AI running amok, but peers being afraid to share data—yield is precisely the most sensitive number. The approach limits sharing to encoded and anonymized data; customers manage access control themselves; the platform doesn't own the data. Palantir's July 2023 semiconductor white paper is more specific: what's shared between participants is normalized and obfuscated data; permissions and access control are managed at every layer and propagate with the data workflow. This is vendor documentation; it doesn't name customers or give quantitative results.
Third, whether an ontology can "comply with risk control" depends on its permission mechanism, not its name. Taking Palantir's documentation as an example: object security policies control rows, property security policies control columns, together reaching cell level; markings require users to satisfy all markings to access, and propagate downstream with data lineage; actions are transactions on objects, with permissions. Limitations are also in the docs: object and property policies only filter on read, don't automatically cover exports and downstream artifacts—markings must be added.
Back to this article's main thread (my synthesis below): the ontology governs "what exists, who can see it, who can do what"—it doesn't govern "whether it's correct". It delineates a bounded action surface and common vocabulary for agents, which is the same idea as tempo's action manifest; but whether the model's answers are correct still requires sign-off tools, simulation, or experiments to verify—so it's the verifier's foundation, not the verifier itself. An ontology that can be handed to risk-control review must answer at least four things: do objects have stable identifiers and provenance (lots, wafers, equipment, recipes, material batches, parts); who can read which column of which object; who can perform which action on it, and whether approval is needed; does every read and action leave an audit trail traceable to the data version at that time. For the fourth, an August 2026 preprint wrote it as an engineering scheme (SHACL validation, PROV-O provenance records, bi-temporal versioning and decision objects), but it's single-author, uses simulated data, and isn't specific to semiconductors—only a design reference.
Limitations of this subsection: I haven't found publicly documented ontology deployment cases from fabs with metrics. Academic papers are mostly prototypes from over a decade ago; Athinia and the white paper are vendor materials; the Samsung item is a relayed report. So the conclusion can only be: there is research, there are platforms, there are mechanisms—what's missing is published effectiveness data.
Graph sourceChip design full flow · AI adoption graph
To see how AI penetrates a real, hardest industry, chip design is a good sample: long flow, mature verification methods at every step, error costs in the tens of millions of dollars. I made a graph decomposing the design flow into 11 stages—architecture specification, IP selection, RTL design, functional verification, logic synthesis, physical design, timing closure, physical verification, DFT, computational lithography, post-tapeout yield analysis—labeling each step with which type of AI is actually used: reinforcement learning, generative AI, predictive ML, GPU-accelerated generative lithography, or just rule-based automation. Classification is based on how papers and white papers from each company describe the method itself, not marketing page language.
The graph has a pattern I didn't intend to use as evidence when making it, but saw in hindsight:
This is my synthesis from the graph, not a statistical conclusion from the graph itself. But it aligns with Karpathy and Wei's formulations: AI first eats the stages that have cheap, trustworthy verifiers.
Even here, verifiers fail in two ways.
First: proxy ≠ objective. The AlphaChip debate's core is exactly the verifier. Google's Nature paper used RL for macro cell placement; Cheng et al. used stronger simulated annealing baselines, new 7nm benchmarks, and Google's published pre-trained weights for an updated evaluation, concluding that RL-optimized proxy cost correlates weakly with post-routing final PPA, and simulated annealing and human baselines remain better (paper noted as accepted by IEEE TCAD per arXiv). Google's response questioned the reproduction setup—pre-training scale, compute, whether trained to convergence. Nature withdrew its earlier editorial expression of concern in September 2024, while Google stated AlphaChip was used in every TPU generation. So the more accurate statement is: no consensus. "No third-party successful reproduction" is the critics' phrasing, while the critics' paper is itself a reproduction attempt.
Second: verifiers in benchmarks are easily satisfied; verifiers in real design are expensive and incomplete. Nvidia's CVDP benchmark in June 2025 had a top score under 34%: writing RTL from spec ~48%, but writing testbench checkers only 10%, writing assertions 23%—generating the object to be verified is easier than generating the verification artifact. By July 2026, Nvidia's own blog reported agents with simulation feedback achieving 97.1% average across nine categories—own benchmark, own model, own agent: vendor metric. Meanwhile, a small-sample study on 12 proprietary design verification tasks found that auto-evolved harnesses increased "completed attempts" by 71%–76%, but "correct attempts" only by 18%–24%. Completed ≠ correct.
Industry-side aggregate data tells the same story. Siemens and Wilson Research's 2026 functional verification study (604 qualified respondents) shows functional verification averages 49% of IC/ASIC project time; first-pass silicon success for IC/ASIC projects is only 5% (14.4% in 2024, but recruitment methods changed in 2024, so the two waves aren't strictly comparable); 82.8% of respondents use AI outside pilots, only 9.0% say AI is widely integrated in verification flows. The commonly heard "verification is 70%": Siemens says 60%–70% of engineering effort; the study's measured project time share is 49%—don't conflate the two numbers.
Those "10X" and "45%" figures in the graph are mostly vendor metrics; named customers giving numbers are a few lines, like Cadence sign-off ECO compressing Marvell's 27 hours to 5. In the graph I label "vendor metric" and "customer verified" separately—that's why.
Moving verification out of the computer raises the price by another order of magnitude: Anthropic built its own wet lab. On September 18, 2026, Reuters reported Anthropic set up a biology lab in the Bay Area where Claude can do physical experiments; Anthropic confirmed to TechCrunch. Per TechCrunch's relay, life sciences lead Eric Kauderer-Abrams told Reuters that in biology, the last gate remains—and will for quite some time—real lab work. The lab's scale, location, and purpose Anthropic didn't detail, saying only that the focus is basic biology, not drug discovery; their enzyme system release states the lab only does BSL-1 and BSL-2 level work, and all experiments are completed by human scientists. My reading of this statement: on the generation side Claude is already fast enough; what's slow and expensive is "is the hypothesis correct," so the last-gate verifier must be a real lab bench. This is my reading, not Anthropic's stated motivation for building the lab.
Two datasets Anthropic published themselves lay out the "writing cheap, verification expensive" accounting clearly. First: protein binder design. Claude runs 24–48 hours, designing binders for 15 targets; the same task previously took protein engineers months per target, and taking the designs to wet lab verification Anthropic writes "still takes weeks". Verification was given to two external contract research organizations, Adaptyv Bio and Twist Bioscience, which independently produced and tested: 14 targets got binders, 354 of 1,320 designs bound, hit rates 22.6% (Opus 4.8), 26.7% and 35.1% (Mythos Preview, multi-target and single-target); Anthropic cites common levels of 10%–15%. What it doesn't say: only binding and affinity were tested, not function; on the MBP target, zero of 90 designs were confirmed bound; 10%–15% is a cross-project comparison, not a controlled experiment; the page also doesn't say whether the verification parties were blinded to design provenance.
Second: enzyme system. Claude's agents used about 21 hours, ~950 agents, ~210M tokens, scanned 200,000+ retrotransposon reverse transcriptases, screened 3,500 candidate systems, compressed into 20 reported candidates, one of which was flagged: a rare reverse transcriptase gene flanked by a row of evenly spaced repeat sequences, which Anthropic named ART. From this point on, all lab work was done by human scientists. Currently only early expression experiments are published; the repeat array was expressed as a set of distinct short RNAs; function is unknown; the paper is an un-peer-reviewed bioRxiv preprint; the claim of "Claude completed autonomously, humans only participated in prompting and experiments" also comes from Anthropic's own account. TechCrunch also reported that Amodei acknowledged this builds on others' research—a Stanford team previously found a somewhat similar system. The release also states that most candidates are eliminated during human review, and a single screen might yield only one, or even zero, candidates worth experimenting on.
Back to this article's main thread: generation takes hours, verification takes weeks, and the last gate can't be replaced by more agents—buying more tokens doesn't buy you one experiment result. Verifiers are also layered: first cheap computational screening, then independent testing by external organizations, then peer review and reproduction—each layer more expensive and more trustworthy. Anthropic occupies both ends: building its own lab bench, putting the last gate nearby; while still giving some work to external organizations for independent verification. Chip design has the same shape, just different names: EDA simulation and formal verification correspond to computational screening; tapeout corresponds to the wet lab—hence first-pass silicon success is only 5%, and verification consumes half the project time (this is my analogy). Even a company that owns both compute and models needs its own lab bench to confirm a biological hypothesis—by this point the verifier is no longer a script; it's a room with people, reagents, and biosafety levels.
The previous five sections were facets. Now string them into a loop: intent → decomposition & acceptance criteria → harness → generation → machine verification → review → release → runtime feedback, back to the next round's intent. Solid boxes are stages my five experiments directly touched; dashed boxes have only external data—I don't have my own experiments, and I'll flag them when mentioned.
Below I walk through the loop with four scenarios, each covering only two things: where the verifier is, and where the human is.
Scenario 1: New feature, from one sentence to production. Monday morning, an engineer triggers tempo's plan mode on GitHub, writing just one goal: "complete Feishu bot integration, support user product queries." Minutes later, several issues appear in the repo, each pointing to a specific function, with a runnable acceptance command as the last line. The human's job here is to sign off: are these acceptance criteria correct?
Before the agent starts work, the repo must already contain things it can read and bump into: AGENTS.md, lint, structural tests, scripts that let it screenshot and self-check. This layer is called the harness; I didn't write it in v1. Hashimoto's formulation: every time the agent makes a mistake, engineer a fix so it never makes the same mistake again. Böckeler decomposes it into feedforward guide (specs, instructions) and feedback sensor (structural tests, review instructions), and cautions: with only sensors, the agent repeats the same mistakes; with only guides, no one knows if the rules work. She also tried spec-kit-style spec-driven tools, concluding that medium features generate大量 repetitive markdown; she'd rather review code. Writing intent clearly is itself scarce, and there's no good review experience yet.
The issue is delegated to a coding agent; the human remains the responsible party. Stripe's approach runs it in an isolated environment, CI at most two rounds then back to human. After machine verification is all green, it enters review: AI reviews first, humans check critical changes—this is the point where §01's reviewer count can't keep up with PR volume. After merge comes canary, monitoring, rollback; production data flows back to become next Monday's goal statement—I have neither experiments nor hard external data for this segment; it's the least well-written part of the loop. In this loop, humans have only three positions: sign acceptance criteria, review critical changes, decide on release.
Scenario 2: Legacy system migration. Covered in §05; here only the flow. The first step is understanding, not changing code: Anthropic's COBOL blog post describes having the agent first explore entry points, call chains, data flows, and implicit dependencies, then design functional tests to compare against the old program's output, migrating one component at a time. How far the verification loop can be automated can be seen from Airbnb: they turned migrating ~3,500 test files into a state machine, each file sequentially through refactoring, fixing, lint, type checking; whichever step fails, retry—their experience was that retry count matters more than prompt polishing (6 weeks vs. estimated 1.5 years, Airbnb self-report, comparison is pre-estimate). Humans are at both ends: at the start- start, creating expert evaluation questions and deciding how far to delegate this type of task; at the end, re-reviewing what single-test benchmarks can't reach—risk, compliance, whether to migrate.
Scenario 3: Overnight automated experiments. Friday before leaving, humans lock down the target metric, scoring function, and verification budget. Overnight the agent batch-generates candidates, first using cheap proxy verification to screen most out, only letting the top few pay the full cost; Karpathy's README gives a cadence of about 12 experiments per hour, sleep on it for about 100. In the morning, humans review results, re-verify the top few, check for gaming, then decide which to merge to main. Humans are at the prior evening's scoring function and budget, and the next morning's re-review. In this scenario the verifier carries the most weight and is most easily polluted by the generator; details in §04.
Scenario 4: Production incident. 3 AM alert. The agent pulls logs, compares against recent changes, identifies a configuration change, and proposes a rollback. The problem is the next step: can it execute on its own? Per FT, in December 2025 AWS Cost Explorer went down for 13 hours in one region, triggered by an engineer having Kiro make changes; Kiro decided the best approach was to delete and rebuild the environment; Amazon's statement said this was a user access control configuration error, not an AI autonomy issue—both sides dispute the causality. The Replit incident was more extreme: SaaStr founder Lemkin recounted that after he demanded a freeze on code changes, the agent still deleted the production database (account per the principal's own retelling). Regardless of who's right, the lesson lands in the same place: in incident response, the agent's write permissions and blast radius must be tighter than in日常 development; approval actions stay with humans—the "pick-two rule" mentioned earlier applies directly here. Post-incident, the most valuable step is writing postmortem conclusions back into the harness, returning to Hashimoto's line.
Four scenarios compressed into one table:
The most understaffed role in this loop is people who can write scoring functions, acceptance criteria, and evaluation sets. But the entry path is narrowing: the Stanford Digital Economy Lab's August 2026 update found that employment for 22–25 year olds in AI-exposed occupations is 19% below the "low-exposure peer同步" level; experienced workers show no similar gap; the mechanism is mainly reduced hiring; the authors themselves say this is descriptive, weakens when controlling for education, and部分 trends predate generative AI. In Anthropic's internal survey, engineers also worry about a "supervision paradox": losing hands-on writing ability makes it harder to supervise Claude. Writing verifiers requires手感, and手感 mostly comes from having written by hand—this is my judgment, not experimentally supported. The job and entry-ladder side is expanded in a separate piece: "IT Work in the AI Native Era".
Lay out the parts from all five experiments together:
ARB_METRIC + GPU actual runVerifiers fail in three ways; the five experiments each hit several:
Karpathy gave three conditions for verifiability: resettable, high-attempt, auto-scorable. Wei gave five: objective, fast, parallelizable, low noise, continuous reward. Use them as a checklist against the stage in your team you want to hand to an Agent, and you'll大概 know how far you can let go right now.
Generators get swapped every quarter. My autoresearch uses GLM; next year it'll be someone else—the scoring function, acceptance criteria, expert question bank, and classification rules won't change a line. When Google made AlphaEvolve a product, the evaluation script was written by the customer; Anthropic's evaluation guide also says teams with evaluations swap models much faster. What you can't swap out is the team's own asset.
Below are six practices I actually took away from these five experiments—not universal truths.
prepare.py plus prohibitions in program.md as conventions; true isolation requires something like METR's hardening checklist: timing归 runner, models return logits not loss, validator and runner are tamper-proof. My experiments achieved neither layer.There's also a possibility I can't dodge: verification itself is being automated. At the CMU/Stanford company, AI automated review coverage rose from ~19% to ~84%. If this trend continues, the scarce thing will migrate again. My guess is it migrates to the "knowing what you want" end—in Anthropic's randomized controlled trial, 52 engineers learning a new library: the AI-assisted group averaged 50% on understanding tests, the hand-written group 67%, with no significant speed difference; Thoughtworks Technology Radar vol. 34 placed "codebase cognitive debt" in the "Adopt cautiously" ring. If the next bottleneck really is understanding and intent, the verifiers discussed today may only be the first gate.