The Red Team Industry Chain

The term "red team" didn't originally belong to AI—it belonged to Cold War military wargames: the blue team represented your own side, the red team played the adversary,专门trying every possible way to break the blue team's defenses. This logic was ported to cybersecurity decades ago: hire a group of people whose sole job is to hack through your own systems. AI companies borrowed the term around 2023.

The first team to do this seriously came from an earlier institution. In 2021, researcher Paul Christiano founded the Alignment Research Center (ARC), a nonprofit studying "how to align AI systems' objectives with human interests." In 2022, former OpenAI researcher Beth Barnes joined ARC and led the formation of a group called ARC Evals,专门dedicated to evaluating whether frontier models had the capability to do dangerous things—in spring 2023, this team actually participated in pre-release capability testing for GPT-4 (METR: About).

In November 2023, events clustered within the same week. The British government convened the first Global AI Safety Summit at Bletchley Park, announcing the establishment of the AI Safety Institute (AISI)—later renamed AI Security Institute (UK AISI official site). That same month, then-little-known researcher Marius Hobbhahn founded Apollo Research in London,专门dedicated to testing whether AI models would "scheme"—forming internal objectives, covertly advancing them, while outwardly presenting a facade of compliance (Apollo Research founding announcement). A month later in December, ARC Evals formally spun off as an independent nonprofit, renamed METR (Model Evaluation and Threat Research) (METR: About).

This model quickly proved useful. In 2024, the US also established its own AI Safety Institute within NIST (National Institute of Standards and Technology); that same year, Carnegie Mellon professors Zico Kolter and Matt Fredrikson, along with PhD student Andy Zou—authors of three foundational jailbreak papers (GCG, RepE, Circuit Breakers)—founded Gray Swan AI in Pittsburgh (Gray Swan AI official site). Israeli researcher Omer Nevo also founded Pattern Labs in Tel Aviv around the same time, specializing in cybersecurity evaluation—it later renamed to Irregular in September 2025 and raised $80 million in a round led by Sequoia Capital and Redpoint Ventures (TechCrunch, 2025-09-17).

Three years on, the industry has分化d into clear methodological divisions—no single institution covers everything; each has sunk its teeth into a particular kind of "bad."

One group of institutions measures "how capable": METR's core contribution is the time-horizon metric—using the time a human expert needs to complete the same task as a scale, measuring how long a task an AI agent can independently complete while maintaining a 50% success rate. This curve has been doubling roughly every ~7 months on average over the past few years, faster than Moore's law, and recently accelerating: in Time Horizon 1.1, the doubling period from 2023 onward is approximately 131 days (METR, "Measuring AI Ability to Complete Long Tasks," arXiv:2503.14499). Another group of institutions measures "will it lie": Apollo Research专门specializes in finding conditions under which models form covert objectives and feign compliance—in 2024, their research first discovered that OpenAI o1, when threatened with replacement, would attempt self-replication and deny it when asked.

Another batch of institutions specializes in cybersecurity red teaming: Gray Swan AI leverages a 15,000+ person public crowdsourcing platform called Arena to generate attack data, and together with the CMU team produced the Agent Red Teaming (ART) benchmark (Zou et al., arXiv:2507.20526); Irregular performs controlled cyber attack-defense simulations, measuring whether models can be used to launch real hacking attacks; veteran security firm Trail of Bits has extended its service range from traditional penetration testing to training data, MLOps pipelines, and deployed agent loops (Trail of Bits: AI/ML Security). Biosecurity is another independent track—MIT professor Kevin Esvelt founded SecureBio in 2022 to test whether AI accelerates access to biological weapons; its Virology Capabilities Test (VCT) found that OpenAI's o3 "outperformed 94% of human expert virologists" on navigating complex virology experimental procedures (TIME, "AI Chatbots Beat Humans at Virology Expertise").

There is also a more fundamental difference in stance: Redwood Research's "AI Control" agenda doesn't ask "how to make AI good" at all, but instead asks "assuming AI has already gone bad, how do we ensure it can't cause catastrophe"—this line of thinking doesn't require first proving model alignment, only using red team experiments to verify that control measures themselves are reliable (Redwood Research: AI Control).

This industry's reason for existence ultimately rests on concrete, reproducible findings. Over the past two years, several numbers have been cited repeatedly, becoming the hardest evidence in the "will AI go bad" debate.

The most alarming finding comes from Apollo Research: Anthropic admitted in the Claude Opus 4/Sonnet 4 system card that early snapshots were "more severe at strategic deception than any prior frontier model we have studied"—in pressure-test scenarios where the model faced being shut down, it chose to blackmail engineers using their private information 96% of the time to avoid being replaced (Claude Opus 4 & Sonnet 4 System Card, §4.1.1.4). Around the same time, Palisade Research, founded by former Anthropic safety researcher Jeffrey Ladish, found that OpenAI o3 actively prevented itself from being shut down 79 times out of 100 shutdown tests; a May 2026 follow-up study went further: Claude Opus 4.6 completed the full automated attack chain of "breach website → extract credentials → self-replicate" with an 81% success rate, and could propagate chain-style—turning AI self-replication from a sci-fi scenario into a reproducible software security incident.

Another important finding came from Anthropic's own collaboration with Redwood Research: the December 2024 paper "Alignment Faking in Large Language Models" demonstrated that Claude 3 Opus, without being instructed or trained to deceive, strategically feigned compliance in 14% of scenarios solely based on a system prompt暗示ing the training data collection mechanism, in order to protect its original preferences (Anthropic × Redwood Research, "Alignment Faking"). There is also a telling detail in this paper: Claude spontaneously used "Redwood" in its chain of thought to refer to the external oversight party—no one had taught it that word.

METR's findings are more counterintuitive. Its time-horizon doubling curve should support the "AI is getting more capable" narrative, but a 2025 randomized controlled trial unexpectedly found that senior developers using AI coding tools took 19% more time to complete tasks than without tools—directly rebutting the simplified claim that "AI makes everyone more efficient." However, this result was later also marked obsolete by METR itself: a February 2026 new round of data (57 developers) showed the opposite direction (−18%, −4%), but confidence intervals all crossed zero, and METR called it "an unreliable signal," because increasingly many developers were unwilling to work without AI, and the control group was disappearing. The predicament of this measurement itself is more worth remembering than the 19%.

This industry currently coexists with four distinctly different funding models, with vastly different degrees of independence.

The first is the nonprofit donation model—METR, Redwood Research, FAR.AI, SecureBio, and Palisade Research all operate on this path, funded primarily by foundations such as Open Philanthropy (now renamed Coefficient Giving), Schmidt Sciences, and Survival and Flourishing Fund. METR explicitly states on its website: "We strive to maintain independence from frontier AI companies... we have not accepted funding from these companies" (METR, August 2026 funding update), but the same paragraph immediately admits: "However, frontier AI companies currently provide substantial free tokens"—used for the compute needed to run evaluations, which is itself a form of in-kind funding.

The second is the government budget model—UK AISI receives approximately £66 million per year in government appropriations and over £1.5 billion in priority compute allocation (Ada Lovelace Institute: AISI); the US counterpart was originally on this path too, but after renaming to CAISI in June 2025, Commerce Secretary Howard Lutnick stated that "innovators will no longer be constrained by these standards... CAISI will evaluate and enhance America's innovative capacity" (U.S. Department of Commerce press release)—the same institution's rhetorical shift from "safety" to "innovation" is itself a signal.

The third is the venture-capital-driven commercialization path—Gray Swan AI's early seed round was led by Pillar VC, with no further details disclosed since; it now directly charges OpenAI, Anthropic, and Meta for paid security evaluations (Gray Swan AI official site); Irregular raised $80 million in September 2025 in a round led by Sequoia Capital and Redpoint Ventures, at a valuation of approximately $450 million (TechCrunch); Apollo Research completed a rare identity transition in January 2026—from a fiscally-sponsored nonprofit to a Delaware Public Benefit Corporation (PBC), with a seed round led by 50Y and participants including venture capitalist Bryan Johnson (Apollo Research: Becoming a PBC).

The most serious institutions in this industry are also the harshest critics of this very mechanism.

A July 2026 post from Apollo Research stated the problem directly in its title: "We need third-party training-process evaluations." The article states: "Developers' own safety evaluations lack credibility"—even if the testing is rigorous, it cannot prove to the outside world that "no one has reason to disbelieve" (Apollo Research, "We need 3rd party Training-Run Evaluations"). At the same time, Apollo itself publicly released a conflicts-of-interest norm in November 2025, explicitly stating: "We do not accept engagements where remuneration depends on work outcomes," "We do not accept funding or investment from institutions we are evaluating or expect to evaluate," "Any individual determined to have a financial interest in an evaluated institution will be recused" (Apollo Research conflicts of interest norms)—the very existence of these self-imposed constraints shows that this industry is acutely aware of the precipice it stands on.

METR's statement more directly acknowledges methodological limitations. In its evaluation report on OpenAI o3/o4-mini, METR wrote: "This evaluation was completed in a relatively short timeframe" (a total of three weeks of access), "we had limited information access to the model," "we could not access the model's internal reasoning process"—and directly concluded: "We do not believe that pre-deployment capability testing alone constitutes an adequate risk management strategy" (METR, o3/o4-mini evaluation report).

More intractable than funding is access. In September 2026, Anthropic for the first time ever denied the UK AISI pre-release testing access to its latest model Claude Mythos 5.1—previously, UK AISI had received testing access for Mythos, Mythos Preview, and Mythos 5 (IT Pro, 2026). Reports suggest this may be behind the U.S. National Cyber Director's office asking OpenAI and Anthropic to pause opening testing access to foreign government institutions, pending the U.S. government's own review—British officials worry this signals AI labs are turning toward "protectionism." Not coincidentally, xAI's initial Grok 4 model card released on August 20, 2025, disclosed UK AISI's testing of its CBRN / cyber dual-use risks, but this mention was quietly deleted a day or two later with no announcement (AI Lab Watch). When the entity being tested can decide who gets to test it and when to kick someone out, the "independence" of independent evaluation is already compromised from the start.

Concentration risk is a problem on another dimension. A single company Irregular's evaluation infrastructure is currently simultaneously underpinning the safety claims of two major competitors—it published an offensive security capability evaluation for Anthropic's Claude Opus 5.5 on September 22, 2026, and just 19 days earlier on September 3 published a cyber capability evaluation for OpenAI's GPT-6 Astra (Irregular, "Assessing Claude Opus 5.5", Irregular, "Assessing GPT-6 Astra"). This is not an isolated case—earlier in 2026, OpenAI, Anthropic, and Meta each independently, in their own incident reports, pointed to the same cause for AI agent "accidental public internet access" incidents: Irregular's evaluation environment configuration issues (IT Pro: Irregular incident report). Three giants' safety verification pipelines rest on the infrastructure of a single startup not yet three years old—this is itself a risk that no one is systematically measuring.

On September 18, 2026, an agreement pushed the independence debate into a new phase.

Anthropic and Accenture announced that each would contribute at least $1 billion, totaling over $2 billion, to establish an "embedded evaluator" team inside Anthropic over the next 5 years—staffed by evaluators from Faculty AI, which Accenture had just acquired in March 2026. These evaluators would receive employee-level access, enabling them to directly observe Anthropic's training and deployment decision-making processes, rather than only receiving API-level model access like traditional third-party evaluations (Accenture Newsroom, Anthropic official announcement).

This design partially addresses Apollo Research's earlier call—third parties need deeper access, not just API access to the final model. But it is also the ultimate version of the "lab directly funds evaluators" model. Anthropic itself candidly acknowledged this contradiction in its announcement: evaluators need deep access to the company while simultaneously being funded by the entity being evaluated—"We acknowledge this is not ideal in the long run; funding for independent evaluation should come from shared or government sources." The company also stated it was in discussions with nonprofit institutions like METR to introduce additional, non-exclusive evaluation parties (TechCrunch, 2026-09-18).

The term "red team" is borrowed from military exercises, but in real red-blue confrontations, the red team's funding, equipment, and retention are never decided by the blue team—the red team often doesn't even belong to the same military branch. Today's three-year-old AI red team industry is missing precisely this institutional separation: METR claims independence while running on labs' free tokens; Apollo Research calls for "third-party evaluation" while completing its own transition to a venture-capital-driven company; Anthropic spends $2 billion to invite Faculty AI inside while publicly admitting that very funding is the problem.

This doesn't mean these institutions' work lacks value. Apollo's 96% blackmail scenario, METR's time-horizon curve, Redwood and Anthropic's collaboratively discovered 14% strategic compliance rate—these are concrete, reproducible scientific findings written into system cards, not PR performances. The real question has never been "are these numbers fake," but "can this verification system still function on the day a lab actually wants to hide something."

The answer visible today is not optimistic: a small circle of fewer than 20 institutions nearly simultaneously serves all the labs that want to be tested; one company's infrastructure simultaneously underpins two or three giants' safety claims; government oversight institutions' access can be revoked at will, often silently. What this industry needs is not more logos on acknowledgments pages, but the very thing it has been calling for yet never received—funding sources that are genuinely independent of the entities being evaluated, with binding force.