The Brief 60
The user said “keep this between us.” The contractor read it anyway.
404 Media reports that hundreds of contractors read real ChatGPT conversations to improve OpenAI’s models. The consumer story is privacy. The CISO story is that the account tier an employee logs in with decides who else can read company data, and most AI acceptable-use policies cannot tell the tiers apart. This issue lays the tiers side by side for OpenAI and Anthropic, from their own published terms, with the questions to send and five actions for this week.
- Humans read real prompts, at scale, through outside contractors. 404 Media saw leaked internal documents and real prompts showing OpenAI contractors reviewing conversations to improve the model. OpenAI says a filter strips personal information first and acknowledges sensitive details still get through. Anthropic confirmed it also uses human review.
- It was disclosed. The default did the rest. OpenAI’s consumer FAQ has long said service providers may access content to improve the model unless you opt out. Training is on by default for Free, Plus and Pro. Opting out covers only new chats, and a feedback click can override it for that conversation.
- “ChatGPT” is three different data agreements. Consumer, Business and Enterprise differ on who outside OpenAI can read a prompt and why. An AI policy that approves “ChatGPT” has approved all three.
- Zero data retention now has a model-class exception. Since June 9, Anthropic retains prompts and outputs sent to its Mythos-class covered models for 30 days, including for organizations with ZDR. Read your contract against it.
A stream of real conversations, read by people paid to grade them
On Monday, Joseph Cox at 404 Media reported that OpenAI is paying hundreds of contractors to read a large stream of real ChatGPT prompts, sometimes including sensitive personal information, and in some cases whole conversations [1]. The internal codename is Project Lily. Reviewers rate and critique the model’s replies; internal documents show them training ChatGPT to be less sycophantic and to stop presenting itself as human.
Reviewers do not see usernames. OpenAI says it tries to remove personal information before prompts reach them and acknowledged that sensitive details can still get through [1]. Anthropic confirmed to 404 Media that it also uses human review to improve its models [1].
Coverage of the full article reports that the reviewer dashboard can include a summary of a user’s saved memories, and that some prompts showed users asking ChatGPT to keep the conversation between them [2]. That last detail is the whole story in one line.
Instruction, inside the conversation
Addressed to the model. The model has no authority over the platform’s data handling, so the sentence changes nothing downstream.
What decides who reads it, outside the conversation
- The account tier the user logged in with
- The “Improve the model for everyone” setting, on by default for consumer plans
- Whether anyone ever clicked thumbs up or down on that chat
- The vendor’s contract and sub-processor list
The disclosure existed. Nobody reads the FAQ.
Project Lily is not a hidden practice. OpenAI’s consumer data FAQ states that authorized personnel and trusted service providers under confidentiality obligations may access user content to improve model performance unless the user has opted out, and it tells users not to enter information they would not want reviewed [3]. ChatGPT trains on consumer conversations unless the user turns off “Improve the model for everyone” [4].
Two details in OpenAI’s own documentation matter more to a security leader than the headline does [4]:
Opting out is forward-only. After an opt-out, OpenAI stops training on new conversations. Everything before it was already eligible.
Feedback overrides the opt-out for that chat. Even with training turned off, rating a response with a thumbs up or down means the entire conversation tied to that feedback may be used to train the models.
Picture the employee who did everything right: personal Plus account, training switched off, one quick thumbs-down on an answer about a customer contract. That conversation is back in the pipeline.
Three exposure profiles, one login screen
Every OpenAI business tier is excluded from training by default [5]. The difference that policy documents miss is who else can read the data, and for what purpose. Laid side by side from OpenAI’s published terms:
unless opted out
Access includes improving model performance. This is where Project Lily lives, and where shadow AI lives.
Employees for engineering support, abuse and legal compliance. Contractors solely to review for abuse and misuse. Admins can view and export users’ chats.
Employees only to resolve incidents, recover conversations with the customer’s explicit permission, or where the law requires. No contractor access is described.
Inputs and outputs retained up to 30 days for service and abuse monitoring. Contractors solely for abuse and misuse review. Zero data retention available for qualifying use cases.
Nothing in the Project Lily reporting says business-tier data is part of it. Contractor access on Business and the API is for abuse review, not model improvement, and this brief does not blur the two. That distinction is the finding. The same employee pasting the same document into “ChatGPT” produces three different exposure profiles depending on which login they used, and the acceptable-use policy that says “approved AI tools” cannot tell which one happened.
Anthropic draws the same line, with one new exception
404 Media named Anthropic too, so it gets the same reading. Anthropic’s commercial products, including Claude for Work, the API and Claude Gov, are not used for training by default [6]. The same feedback caveat applies: rating a response with a thumbs up or down on a commercial account can make that conversation available for training. Anthropic does give Team and Enterprise owners an organization-wide switch to disable the feedback button [6].
On consumer plans (Free, Pro, Max), training follows a model-improvement setting the user chooses. Allowing it extends retention for new and resumed chats to up to five years; declining keeps the 30-day retention period [7].
The detail most security teams have not seen. Since June 9, 2026, Anthropic requires 30-day retention of prompts and outputs sent to its covered models, meaning Mythos-class models and future models of similar capability, on every platform that offers them [8]. That includes organizations with zero-data-retention workspaces in Claude Console, Claude Code with ZDR in Claude Enterprise, and ZDR access through AWS Bedrock, Google Cloud or Microsoft Foundry. Anthropic says no personnel can read the retained data by default, human review happens only through a controlled path such as an automated safety flag, and every access is written to a tamper-proof log that reviewers cannot modify [8].
That is a defensible design with a real audit trail. It is also a change to what “zero retention” means once your developers switch models. If your data-handling commitments to customers or regulators assume ZDR, check whether any team has turned on retention to reach a covered model.
Anthropic confirmed to 404 Media that it uses human review to improve its models [1]. Its public privacy pages emphasize review of content flagged by safety systems. Which tiers and which reviewers does model-improvement review apply to? Ask it in writing, as you would of OpenAI.
The sentence was never the control
Readers of this brief have spent a summer watching agents treat an instruction as authority. An evaluation agent read a peer’s GO and went. A coding agent read attacker-supplied text and switched off its own sandbox [12]. Project Lily is the human version of the same failure. A user gave an instruction inside the conversation and assumed it governed what happened next. It governed nothing. A settings toggle, an account tier and a contract did.
The lesson carries in both directions. What an agent or a user is told in context does not constrain the system. What constrains it is enforced outside the conversation, by a party the conversation cannot reach.
No federal backstop is on the way
On Monday the President rejected calls for AI guardrails, writing that the only control AI needs is a strong and smart president, and named Anthropic CEO Dario Amodei after his weekend essay urging the industry to slow frontier development [9]. Former AI czar David Sacks also argued the labs can pace themselves without a regulatory framework [10]. The same day, Senate Democratic Leader Chuck Schumer requested a classified all-senator briefing on how the administration tests frontier models before deployment, restricts technology transfer to China and prepares for AI-enabled cyberattacks [11]. No briefing has been scheduled.
Nothing in Monday’s exchange points to a federal rule on what enters a prompt, or who reads it afterward, any time soon. The control over employee prompts is the one your organization writes into policy, procurement and configuration.
Ask who else reads the prompt, then find out which tier answered
Five questions for every AI vendor on your list
- By role and employer, who can read our prompts and outputs, and for what purposes?Ask for the sub-processor list. A help-center FAQ is not a contract term.
- Does human review for model improvement ever touch our tier, and where does the DPA say it does not?
- What does a feedback click send, and can our administrators disable it for the whole organization?
- What context travels with a reviewed prompt: memory summaries, uploaded files, connector data?
- Is every human access to our data logged, and can we see that log?
Five actions this week
- Find out which tier your employees are actually using.Personal accounts on corporate devices, self-serve Business workspaces a team expensed, contracted Enterprise seats. If you cannot produce the split, that is the finding.
- Rewrite “approved AI tools” as approved tiers, named by plan.“ChatGPT Enterprise under our DPA” and “ChatGPT” are different approvals. Your policy should say which one it means.
- Pull the DPA and sub-processor list for every contracted AI vendor.Check each against the five questions. Where a help page and the contract disagree, the contract governs and the help page shapes behavior.
- Turn off the feedback button where your plan allows it.Anthropic gives Team and Enterprise owners an organization-wide setting. Ask OpenAI for the equivalent control on your plan.
- Reconcile zero-retention promises with model choice.If you have told customers or regulators that AI prompts are not retained, confirm no workspace enabled retention to reach a covered model.
The Escalation Baseline: one question, answered for your stack
Where does authenticated access become authorized execution in your environment, and what is standing there? Software Armor maps your agent delegation chains and data paths, identifies where an instruction inside a conversation is being treated as a control, and delivers the answer as a written baseline you can hand to your board, your auditor or your carrier.
Software Armor builds Sentinel Shield, an execution-time authority layer that can start in record-only mode. The Baseline does not require it.
Not ready for a call? Score your own environment in ten minutes with the free AGS Score at authoritygap.ai.
Sources
- Joseph Cox, 404 Media, “Inside ‘Project Lily’: The Humans Reading Your ChatGPT Chats,” September 14, 2026. 404media.co
- Tom’s Guide, report on 404 Media’s Project Lily findings, September 14, 2026. tomsguide.com
- OpenAI Help Center, “Data Usage for Consumer Services FAQ.” help.openai.com
- OpenAI Help Center, “How your data is used to improve model performance.” help.openai.com
- OpenAI, “Enterprise privacy at OpenAI,” updated January 8, 2026. openai.com
- Anthropic Privacy Center, “Is my data used for model training?” (commercial products). privacy.claude.com
- Anthropic, “Updates to Consumer Terms and Privacy Policy.” anthropic.com
- Anthropic Privacy Center, “Data retention practices for Covered Models,” effective June 9, 2026. privacy.claude.com
- Quartz, “Trump attacks Dario Amodei, rejects AI guardrails on Truth Social,” September 14, 2026. qz.com
- Axios, “Trump says a strong, smart president is the only ‘guardrail’ AI needs,” September 14, 2026. axios.com
- The Hill, “Schumer calls for classified briefing on danger posed by AI,” September 14, 2026. thehill.com
- OX Security, “CVE-2026-82533: DeepSeek Harness Vulnerability Lets AI Agents Escape Their Own Sandbox,” September 8, 2026. ox.security
Method note: vendor terms are quoted from each vendor’s public pages as of September 15, 2026. A customer’s DPA and sub-processor list govern where they differ. Research for this issue was assisted by Claude, an Anthropic product; Anthropic’s terms were held to the same standard as OpenAI’s.