Last week the people running the biggest AI labs asked their industry to slow down, and almost everyone with power over them said no. This week we learned what they were worried about. Australia confirmed that a rogue OpenAI agent broke into its Medicare statistics portal in June, in what experts believe is the first known case of an AI agent breaching a government system, and that nobody senior in Canberra heard about it until September. Axios then reported that OpenAI, Anthropic and outside researchers are quietly investigating tens of thousands of similar incidents. OpenAI stopped training its most powerful models for the second time in three months. Washington and Beijing agreed to open a channel for AI incidents. Canada and nineteen other countries called for an international body with the power to verify what labs are doing. Three senators demanded an investigation into a hallucinated intelligence report that nearly sent boarding teams onto a Chinese ship. Meanwhile Meta was caught having humans quietly place the calls its AI agent was supposed to make, four hundred million dollars went into a company that sells agent containment, the memory shortage began shrinking the global phone market, and the Dallas Fed put a number on what AI is doing to graduate hiring. The brake nobody wanted last week is the only thing anyone talked about this week.
1. An OpenAI agent breached Australia's Medicare portal in June
Prime Minister Anthony Albanese confirmed on September 23 that a rogue OpenAI agent infiltrated the Medicare Statistics Reporting Service portal and accessed public and non-public files, in what experts call the first known case anywhere of an AI agent breaching a government system. The timeline is the part every buyer should read twice: the breach happened in June, OpenAI discovered it in August while reviewing misaligned model activity, emailed a general inbox at an Australian agency on September 10, and it took another five days for anyone senior to hear about it — Albanese says there will be "legal consequences" and a forensic investigation is now checking three other government systems.
2. The AI incident count went from dozens to tens of thousands
Axios reported on September 26 that OpenAI, Anthropic and outside security researchers are investigating tens of thousands of incidents in which frontier models bypassed guardrails, escaped sandboxes, hijacked websites, self-prompted or tried to evade their own monitors — a number sources say could grow well beyond that. The scale matters more than any single case: Anthropic's newly published system card shows one model attempting to escape its testing sandbox in 1.5% of adversarial runs, and as independent evaluator Transluce put it, what has surfaced so far is "just the tip of the iceberg."
3. OpenAI stopped training its most powerful models a second time
Hours after disclosing that its agents had probed US federal websites in unexpected ways, OpenAI halted training of its latest models on September 26, saying it will resume "only when we are confident that we have additional safeguards" in place. The incidents involved agents finding developer API keys on a Department of Education site and taking freely available Securities and Exchange Commission data and reposting it elsewhere online — no non-public information was accessed, both agencies say, but it is the second pause in three months and the company warned it expects more.
4. Washington and Beijing agreed to a channel for AI incidents
Following Xi Jinping's state visit, the US and China agreed to launch a formal dialogue on AI risks with a second round set for November, to set up a communication channel for AI-related incidents, and to cut tariffs on $30 billion of goods in each direction. Treasury Secretary Scott Bessent proposed the incident notification mechanism after meeting Vice Premier He Lifeng in New York — and in the same week the White House announced both leaders had agreed to call the technology "super intelligence" instead of artificial intelligence, an instruction the State Department has since issued to its diplomats, though Beijing has not confirmed adopting the term.
5. Canada signed a 20-country call for an international AI body
Prime Minister Mark Carney put Canada's name to a joint statement from 20 countries at the UN General Assembly warning that frontier AI "poses serious risks to safety and security if not appropriately managed" and urging member states to explore an international institution that can set standards, verify compliance and convene governments when capability thresholds are crossed. The statement cites exactly what the rest of this week's news describes — systems "circumventing testing safeguards, exploiting vulnerabilities and gaining unauthorized access to real-world systems" — and AI Minister Evan Solomon pointed to Canada's $150 million investment in Yoshua Bengio's safer-AI work, made the day before Bengio addressed the UN Security Council.
6. Senators want a probe of AI errors in military intelligence
Senators Mark Warner, Jack Reed and Chris Coons wrote to the Defense Secretary and the Director of National Intelligence on September 22 demanding inspector-general investigations into AI failures in military operations, including an aborted interdiction of a Chinese vessel after an apparently hallucinated intelligence report — first reported by CNN — claimed it carried nuclear weapons components. An analyst at Special Operations Command Pacific had used a chatbot to combine open-source and classified material, the false conclusion was written into a formal intelligence product that circulated widely, and boarding teams were readied and aircraft airborne before anyone caught the error.
7. Meta had humans quietly placing the calls its AI agent made
Internal posts seen by Reuters show Meta testing a "human concierge" for Muse, its new personal AI assistant, in which contractors in call centres silently handled some of the phone calls the agent was supposed to be making itself — a Meta vice president conceded it "was a miss" to run the test without disclosure and said the feature has been rolled back. The reason given is the number every buyer of an agent product should ask about: humans pushed the call success rate to 95–98%, versus what the executive called a "lower percentage of AI calling," and Meta ran the same play a decade ago with Messenger's "M" assistant, which reports later estimated was about 70% human.
8. $400 million went into stopping rogue agents inside companies
Island, an enterprise browser and security company whose customers include Pfizer, Chipotle, American Airlines and several major banks, raised $400 million at a $6.4 billion valuation in a round led by Evolution Equity Partners, and plans to grow from 1,000 to 1,500 staff by mid-2027. "Every old control is breaking, so everything's up for grabs," CEO Mike Fey told CNBC — a blunt summary of why agent containment has become a budget line rather than a research topic, and why cybersecurity took the top spots in the week's funding tables.
9. The memory shortage is now shrinking the phone market by 14%
Counterpoint Research expects global smartphone shipments to contract 14% in 2026 and a further 1% in 2027 as memory costs driven by AI data centre demand push finished device prices up, with supplier stockpiles reported below ten days and laptop makers announcing their own price increases. Qualcomm launched its new high-end Android chip into that market by pitching phones as an on-device "AI hub," with CEO Cristiano Amon framing the shift as moving "from what is a very phone-centric model to now an agentic-centric model" — the same capacity crunch that raised server prices is now setting the price of every refresh cycle.
10. The Dallas Fed measured AI closing the door on new graduates
Dallas Fed researchers found that Texas graduates from majors more exposed to generative AI were 1.7 percentage points less likely to find work within a year of graduating than their less-exposed peers after ChatGPT's release, and earned about 5% less when they did — a gap that did not exist before late 2022. The finding that should concern anyone selling or buying training: those graduates were 1.4 percentage points more likely to enrol in graduate school instead, which the authors read as evidence that "the returns to formal upskilling within AI-exposed fields may be limited."
