You can do everything right and still get breached — if the vendor holding your data doesn't. Notion disclosed Wednesday that a breach at Fireworks AI, which runs some of Notion AI's open-weight models, exposed Notion AI chat transcripts from a small number of workspaces. Notion's own systems were never touched. Its users' words were sitting in someone else's dataset.
The timeline, according to BleepingComputer, is a slow-motion reveal. Fireworks alerted Notion on November 2 about an exposed dataset it said held only synthetic test data. Notion cut off traffic, deleted its data from Fireworks' systems, and halted training runs by November 3. On November 6, Fireworks said customer data may have been accessed; on November 8 it confirmed it — including Notion users. Notion began notifying affected customers on November 9, and on November 11 it revoked Notion API keys found pasted into the exposed messages.
The transcripts came from the Insights dataset — which shows admins how teams use Notion AI — which Notion has now disabled. For the vast majority of affected workspaces, Notion says the chats were "unattributable": not labeled with who wrote them or which workspace they came from, with no tool-call results. Fewer than 1% of users are affected. But unattributable is not harmless — a chat with no name on it can still hold a pasted password, a customer list, or a contract. Notion is telling affected users to rotate any passwords, API keys, or credentials entered into those chats.
The sharpest detail is the one Notion put in its own FAQ. Fireworks had contractually committed to zero data retention for Notion's data — and during the investigation, Notion learned the data had been retained anyway. This is the second Fireworks incident in about a month: hackers got into API keys and tokens in early October, and on October 23 a third-party cloud provider hosting Fireworks' research machines was hit. Fireworks links the attack to a crew that sells unauthorized access to language models, known as LLMjacking. Notion no longer serves open-weight models through Fireworks.
Every AI feature is a supply chain. Your prompt leaves the app you trust and lands in a vendor's infrastructure, a sub-processor's cloud, an analytics dataset you never asked about. A zero-retention clause is only as good as the vendor's enforcement — and here it failed twice in a month. Watch whether other Fireworks customers disclose downstream exposure of their own, and whether Notion's investigation widens beyond the 1% it has named.
— via BleepingComputer