Data retention vs. the append-only evidence chain
How Audact's append-only evidence chain and GDPR-style content deletion coexist through crypto-shredding of PII encryption keys.
The append-only evidence chain and a GDPR-style content deletion schedule are not in conflict, because the chain never stores deletable content in the first place.
Span encryption replaces redaction; the chain remains append-only
evidence_eventsis genuinely append-only, enforced at the database level, not just by application code: database rules turnUPDATE/DELETEon the table into no-ops, plus a trigger checks each new row'sprevious_hashagainst the prior row'shashon insert (hash-chain integrity, checked at write time).- Personal data spans are encrypted, not redacted. Every span detected by the
pii_scandetectors (BSN, IBAN, card, email, phone, DOB, postal address) is replaced in persisted copies by a token⟦pii:v1:<TYPE>:<keyset_id>:<base64url(nonce‖ciphertext‖tag)>⟧, preserving the surrounding text as readable. The tokens are sealed before the event is hashed and written, so the ciphertext token becomes part of the hashed evidence data. - Evidence summaries record which pattern/phrase matched and may render the token (dashboards
show
[Encrypted Email]chips); the raw transcript or phrase text never rides along in plaintext. - Consequence: erasure never requires touching a single
evidence_eventsrow. The operator shreds a data key viaPOST /v1/pii/shredorPOST /v1/calls/{id}/pii/shred, and existing tokens become unreadable without modification. The hash chain, and the Merkle/RFC-3161 timestamp anchoring built on top of it, remain intact and tamper-evident regardless of whether the keys exist.
Identities and key material are stored wrapped; only tokens are persisted in evidence
- Identities: Each distinct plaintext value (a name, email, phone number) is tracked as a
separate identity in
pii_identities, with its own random 32-byte data key. The plaintext value itself is never persisted or logged — only itslookup(HMAC-SHA256 of the type and client hash) is stored, allowing the tenant to find it without plaintext. - Key wrapping: Data keys are wrapped with the tenant's
K_wrap(from the tenant's keyring in Secrets Manager) using AES-256-GCM before being written topii_identities.wrapped_key. The wrapped key can be decrypted only by the tenant; shredding an identity NULLs its wrapped key, making the identity's data permanently inaccessible. - Keysets: Tokens are sealed under content keys derived from the identities involved in the
call. Each distinct set of identities gets a short
keyset_idinpii_keysets. Shredding any identity in a keyset makes every token sealed under that keyset unreadable. - Erasure is cryptographic: No background job scans for expired rows. Instead, the operator
(or tenant via the API) calls
POST /v1/pii/shredwith an identifier's client hash. The service finds the identity, NULLs its wrapped key, and publishes an eviction on Redis pub/sub. Existing tokens become[ERASED_<TYPE>]when opened, and the evidence chain shows aPII_SHREDDEDevent proving when the erasure occurred.
Erasure lag: when is data actually unrecoverable?
| Copy | Lag until unrecoverable |
|---|---|
| Live database tokens + service key caches | ≤ 60 seconds (after a shred, keys are cached in-process with a TTL) |
| Tenant keyring | Secrets Manager recovery window (30 days production / 14 days other) + mandatory hold period (35 days) |
| Redis webhook streams | Tokens are sealed at emit; streams are trimmed after 7 days |
| Webhook dead letters | Tokens; purged at expires_at (90 days default) |
| Database backups | Wrapped data keys in pii_identities are encrypted. On restore, previously applied shreds must be re-applied manually using the pii_shreds audit log |
Forward-only cut-over; recordings are excluded
- Cut-over: As of this release, new calls have encryption enabled by default and all detected spans are encrypted. Calls started before this release may carry plaintext spans in their evidence (those rows are hashed into the chain and cannot be edited); they cannot be retroactively encrypted.
- Recordings: Audio recordings are stored in S3 with SSE-S3 encryption and versioning, backed by a 7-day lifecycle lock and separate retention policies. They are not covered by crypto-shredding in this release; erasure of a caller's identity does not erase the recording. This is tracked for a future milestone.
- Names: Name identities exist only if the tenant supplies them as hashes at call init or via the identity-link API. Names spoken in a conversation are not auto-detected, so a name erasure is weaker than email/phone erasure — it erases only the identities the tenant registered, not every mention in the call. The API documentation covers this limitation.
Crypto-shredding delivers immediate erasure; recordings are separate
As of this release, crypto-shredding is the primary GDPR Article 17 erasure mechanism for PII in evidence. It achieves erasure immediately:
- The tenant (or operator) calls
POST /v1/pii/shredwith an identifier's client hash. - The service finds the identity, NULLs its wrapped key, and publishes an eviction on Redis pub/sub.
- Within 60 seconds, existing tokens encrypted under that identity become unreadable.
- An
[ERASED_<TYPE>]placeholder is returned on any subsequent decrypt attempt, and the evidence chain shows aPII_SHREDDEDevent.
Recordings remain a separate issue tracked for a future milestone:
- Call recordings (audio files in S3) are not covered by this milestone.
- They are stored with SSE-S3 encryption and versioning, backed by a 7-day lifecycle lock and separate retention policies.
- Erasure of a caller's identity does not erase the recording.
- A future release will implement:
- A runbook for out-of-band recording deletion.
- Possible future integration of recording lifecycle with identity shredding (requires storage I/O architecture decisions).
