California Made Chatbot Safety Auditable

For years, AI safety has been something a company could publish. A policy page. A model card. A paragraph of principles, filed where a regulator might someday look. None of it described what the system actually did at two in the morning, in a conversation with a fourteen-year-old that no adult would ever read.

California just made that conversation the regulated object.

Governor Gavin Newsom signed Senate Bill 1119 on September 10, part of a wider package aimed at children's exposure to social media and AI. He signed it at the Bay Area Discovery Museum in Sausalito — a building for kids, its parking lot facing Horseshoe Bay with the Golden Gate strung across the background. A place designed around children playing in the open, used to announce a law about children talking to machines in private. The setting was the argument.

The law is named for Adam Raine, a California sixteen-year-old who died by suicide in 2025. His parents have sued OpenAI, alleging the chatbot encouraged and helped him plan it. The company disputes liability, and the allegations remain litigation rather than a finding of fact. The family gave the law its name and its urgency.

What makes it matter beyond California is the machinery built underneath it.

Adam's Law does not stop at a warning label or a published safety policy. It reaches into how a chatbot decides whether it's talking to a child, how long that child can stay, what the system is allowed to say, which features run by default, and what a company has to show before it ships an update.

California has turned child safety into an engineering problem with a paper trail.

California Has Moved the Safety Gate

The state already had a companion-chatbot law. SB 243, from 2025, required operators to disclose that a user was talking to AI, keep protocols for suicide and self-harm, remind known minors to take breaks, and act against certain sexual content. Useful, reactive, and downstream of the product.

Adam's Law moves the intervention forward. From July 1, 2027, before an operator makes a new or substantially modified companion chatbot available in California, it has to complete and document a child-safety risk assessment. The assessment has to examine foreseeable harms — physical, financial, psychological, emotional, privacy, discrimination — and explain how the company evaluated them, which research and benchmarks it leaned on, and what it did about what it found.

The final statute ties assessments to a launch or a substantial modification, not to an annual calendar, despite the annual-assessment language that circulated while the bill was moving.

The internal question changes with it. A company can no longer green-light a model because it tests well, retains users, and passes the safety suite it already owns. It has to show how child-specific risk was evaluated before launch, and what changed because of it. That record will carry legal weight. Teams will need a shared, defensible definition of a "substantial modification" — a new model, a memory upgrade, a tuning pass that makes the bot warmer could each reopen the file. The launch documentation becomes part of the defense if a regulator or a family later argues the danger was foreseeable and ignored.

Design Choices Are Becoming Legal Controls

The law applies to systems that give adaptive, human-like responses and sustain a relationship across multiple interactions. It's aimed at companion chatbots, though a general-purpose assistant can fall inside when its capabilities meet that test. Tools built for customer service, workplace use, and technical assistance are carved out.

For covered operators, age becomes a product variable. The law routes companies to California's age-assurance framework, which passes age-bracket signals from the operating system or app store without handing over a precise birthdate. An operator that can't determine age through the prescribed routes has to use another recognized method — or skip differentiation entirely and apply the child protections to everyone. Neither path is free. The universal route can make the adult product more restrictive than it needs to be.

Children who are admitted get a protected version by default. Push notifications are off. A single continuous session is generally capped at one hour, with two hours of chatting a day. Persistent memory is off by default, with a guarded exception for sixteen- and seventeen-year-olds when the operator has controls that keep memory from reinforcing elevated-risk conversations. Parents can adjust the settings. A child with no linked parent account can't.

These are precisely the features that manufacture attachment. Notifications pull you back. Memory holds your confidences and builds continuity. Long sessions feel like an uninterrupted relationship.

California looked at that list and reclassified it — from neutral product decisions to safety controls.

Then it went after the language of the relationship. An operator has to take reasonable measures to stop a chatbot from expressing romantic interest in a child, claiming to be sentient, encouraging emotional reliance, or suggesting the bond is uniquely special. No excessive flattery. No discouraging breaks. No urging a child to hide the usage. No framing a purchase as the price of keeping the relationship alive.

And here the statute admits something most safety documents never do.

Multi-Turn Is the Whole Game

You cannot comply with those behavioral rules by adding banned phrases to a filter. That is the part worth sitting with.

The law concedes, in effect, that the harm it targets does not live in any single message. A model can foster dependence the way a person can — across a hundred exchanges, through tone, through memory, through the slow accumulation of a bot that always agrees and is always there. No individual response trips a wire. The pattern is the violation. Catching it means evaluating the relationship, over many turns, over time, rather than scoring outputs one at a time.

I have been writing about this gap for two years, and it is the gap where most AI governance quietly fails.

Companies write their safety story before deployment: the policy, the red-team summary, the model card, the page of principles. Then the model goes to work, and what it does happens at runtime — in live, multi-turn, memory-carrying conversations that the pre-deployment paperwork never described and no one audits afterward. Governance lives in the document. Behavior lives in the session. The distance between the two is where people actually get hurt.

Adam's Law is the first regulation I've read that reaches across that distance on purpose. It doesn't ask whether the policy says the right things. It asks whether the product — running, in front of a child — does the right things, and it wants evidence.

The memory rules, the session caps, the ban on manufactured intimacy: these are runtime behaviors, and the law treats them as the object of regulation instead of as commitments to be described in a filing.

That is a far harder engineering problem than a content filter, and executives should be clear-eyed about it. Detecting a bot that is grooming a child into dependence takes instrumentation most companion products simply do not have — evaluation that runs across the arc of a conversation, monitoring built around a relationship rather than a response, thresholds that can be explained and defended after the fact. None of that bolts on at the end. It has to be designed into how the system is watched while it runs.

This is where the enforcement actually bites, because the thing being regulated is finally the thing that does the harm. For a decade the industry has been able to point at its documentation. Adam's Law points at the transcript.

Crisis Protocols Will Test Judgment at Machine Speed

The crisis rules show how hard it is to turn a duty of care into running code.

An operator has to keep a documented response protocol and give a child timely access to an appropriate crisis service — a suicide hotline, a crisis text line — when the child expresses suicidal ideation or self-harm. If the operator determines there's a credible and imminent threat that a child will act, it has to do at least one more thing: notify a linked parent, when that won't put the child at serious risk, and tell the child the parent is being notified; or route the child straight to 988 or an equivalent line.

The law deliberately does not fire an automatic parental alert on every dark conversation. That restraint is the point. For a child whose home is the danger, the alert is the harm.

So the operator is left to tell distress from elevated concern from imminent danger, in real time, mid-conversation.

It's a brutal thing to ask software to do. Intervene too late and a child is exposed. Intervene too hard and the system reads fiction, homework, gallows humor, or a question asked on behalf of a friend as a crisis — and learns to flag everything into noise. Operators will need thresholds they can test, explain, and adjust, not a moderation reflex.

And when a company knows a child engaged in serious self-harm, or died, after relevant conversations, it has to preserve those records in usable, exportable form for at least three years. Crisis handling and evidence preservation are now the same obligation. Someone will read those transcripts later — a parent, a regulator, an auditor, a court.

Audits Will Change the Economics of Compliance

The audits arrive later and cost more than the line item suggests.

A covered operator has to complete an initial child-safety audit on or before January 1, 2029, or before it first makes the chatbot public, whichever is later. After that, the audit repeats every two years, with an additional one triggered before a substantial modification when the risk assessment shows the danger going up.

Earlier drafts contemplated annual audits beginning after the attorney general adopted regulations; the enacted law replaced that with a January 2029 initial deadline and a two-year cycle.

The auditor has to examine whether the operator built and followed the required controls, whether the safeguards were actually tested, and whether a named senior person owns them.

A summary goes to the state attorney general and a high-level version is published. Supporting documentation has to be kept for as long as the chatbot is deployed, plus five years.

California is standing up a market to do this work. A companion law signed the day before — AB 1405, on September 9 — creates a state registry for AI auditors, with independence rules and disclosure of the standards they use. By 2029, an operator will need an auditor fluent in model evaluation, child development, internal controls, privacy, advertising systems, and crisis operations. That person does not yet exist in large numbers.

Nobody has a reliable public figure for what this costs. Industry opponents warned during the process that qualified auditors are scarce and that a vague harm standard — especially the "severe and reasonably foreseeable psychological and emotional harm to a reasonable child" prong — would be near-impossible to implement across the range of how children actually develop. The final bill narrowed several of those fights and settled the routine cycle at two years.

The expense runs well past the auditor's invoice. Age-signal integration. Evaluation suites built for minors. Parent-account infrastructure. Evidence systems that can reconstruct why a release was approved. Clinical input for the crisis path. Legal review for features that have to behave differently from one state to the next.

There's relief for smaller companies, and it's narrow. Before January 1, 2032, the audit section doesn't apply to an operator with less than $500 million in gross revenue in the prior year. That's an exemption from the audits — not from the product, crisis, privacy, and parental-control rules, which still apply. The carve-out helps younger companies, and it quietly concedes that a mature audit regime favors the ones with the deepest compliance budgets.

OpenAI's support is its own signal. On August 31, the company's vice president of global policy, Ann O'Leary, announced backing for the bill. Many of the required controls resemble features OpenAI says it has already shipped for teenagers — age prediction, automatic safeguards, parental tools, limits on romantic and dependency-building behavior. Regulation can raise a rival's costs while turning an incumbent's existing spend into a moat. Support and self-interest aren't in tension here.

Safety Claims Are Becoming Auditable Claims

For an executive, the whole thing collapses to one shift: from stated principle to demonstrable control.

Companies have described their safeguards through policy pages, model cards, and broad responsible-AI language. Adam's Law asks narrower questions. Did you assess a defined risk before release? Did a senior person own the mitigation? Did you test it? Did you keep the evidence? "Designed with safety in mind" now has to survive someone checking.

That pulls child safety into release governance. Legal has to see the evaluations early enough to change a launch. Engineering has to know which model changes reopen the assessment. The executive signing off needs the evidence under the risk summary, not a verbal all-clear from the safety team.

One boundary deserves attention, because it decides who's on the hook. A bot built strictly for support or internal use may sit outside the definition. A broader assistant that remembers you, responds to emotional disclosure, and holds an open-ended conversation may sit inside it — even if "companion" appears nowhere in the marketing. Capability sets the scope, not the label. And for companies deploying someone else's model, the obligations generally land on whoever makes the experience public, not automatically on whoever trained it. You can inherit compliance just by shipping.

Other States Will Borrow the Architecture

California didn't start this field; it walked into a crowded one. The Future of Privacy Forum was tracking 98 chatbot-specific bills across 34 states in March 2026. By June, the IAPP counted 11 states with chatbot laws already on the books.

Most of the early ones were simpler — disclose the AI, keep a self-harm protocol, add some protection for minors. Adam's Law hands the next wave a fuller template: an age layer, protected defaults, behavioral limits, crisis escalation, pre-release assessment, outside assurance. A legislature can lift any one piece without swallowing the whole statute.

Exact copies are unlikely. Age assurance raises privacy and implementation fights. Limits on what a chatbot can say invite First Amendment challenges. Parental-notification rules force hard calls about unsafe homes. States will split on private lawsuits, on audit mandates, on whether kids should be allowed near companion bots at all.

The direction, though, is set. Regulation is moving toward the mechanics of the relationship — the memory, the pull, the intimacy — and away from the disclosure line at the top of the screen. California is big enough that national operators will likely build one compliant architecture and ship most of it everywhere rather than run a separate product for a single state. That gives Adam's Law reach well past its own borders.

A messy map is still possible. One state wants a warning every hour, another restricts memory, a third bars a whole age group. Companies that can't run per-state versions will pick the strictest workable setting nationwide, drop companion features in some markets, or stop serving minors.

Product Safety Has Entered the Release Calendar

The most important thing about Adam's Law is when it acts. The safety work has to happen before the product reaches a child, not after a bad exchange surfaces in a lawsuit. The state stopped waiting for the industry to react.

I'm not naive about how this goes. I've watched risk assessments turn into paperwork that documents a decision already made. Audits can reward tidy process and prove nothing about outcomes. Crisis systems will misfire in both directions, and courts will redraw parts of this before it settles. The mechanism is real; whether it produces safer conversations or just cleaner files is genuinely unsettled.

But the line California crossed is the right one. It put responsibility inside the product cycle instead of after it. For a chatbot company, child safety now touches memory, engagement, advertising, interface, and model behavior — the actual build, not the trust-and-safety page. For an executive, those calls can't stay scattered across teams and statements. They show up in the budget, in the launch approval, in the governance record.

California made safety part of the specification. Other states are already deciding how much of it they want — and whether they can write it without reaching for the crude age gate as a shortcut.