Backups and restore

The data directory holds every credential your people have connected, every receipt, every policy, the audit record, and the collections a job produced. One node means one copy of all of it, so the backup is not an optional extra: it is the availability story.

Two things have to be true before you rely on the deployment. Backup is continuous, to storage that is not this machine. And the restore has been done, by you, at least once, onto a fresh machine, ending with a check that the restored node actually holds rows.

What is backed up, and how

Each module keeps its own SQLite database under the data directory. The bundle ships a continuous-replication sidecar configured against every one of them, shipping write-ahead frames to any S3-compatible target as they are written. The recovery point that buys is seconds rather than the age of last night's snapshot.

The configuration names every database file explicitly rather than matching a pattern, because a database missing from the list is a database with no backup and nothing that says so. A build gate compares that list against the databases the server actually opens and fails when the two diverge, so a version that adds a database cannot ship one that goes unreplicated.

Give the sidecar its own credentials, scoped to one private bucket, with read and write and nothing else. It names each database as replication starts, and that log line is what tells you the coverage is complete.

What is not backed up

Two things under the data directory are not databases and the replicator does not carry them.

Collection rows. These are what a bulk job produced. They are reproducible by re-running the job, and a project that wants them durable in the first place should connect a storage backend so they rest in object storage rather than on the node's disk.

Program modules. These arrive with the release or through the management API rather than being written by the node.

If either becomes something you must preserve, add a nightly sync beside the replicator rather than trying to make the replicator carry it.

The vault key is not in the backup, deliberately

Connection credentials and signing secrets are sealed with your vault key before they reach a database file, so the replicated objects are useless to whoever holds the bucket alone. Storing the key beside the backup would undo exactly that.

Keep the key in your secret manager, backed up on its own path, with its own access list. The identity database is the one file whose secrets are hashes rather than sealed rows, so it restores usefully without the key. Nothing else does.

The restore drill

Run it when the backup is first configured, and again after any change to the bucket or the credentials.

1. Restore every database from the bucket into a scratch directory on a machine that is not the node.
2. Start a throwaway server against that directory, with the vault key present, on a loopback port.
3. Probe it. A node that answers has opened the files.
4. Read something out of it. automaton apikey list -data <scratch dir> is the cheapest proof, and it is the step that matters: a restored database that opens with no rows passes every step above it.
5. Connect one account in the throwaway node, or confirm an existing connected account still works. That is the check that the vault key you restored with is the key those rows were sealed under.
6. Tear it down.

Recovering the real node is the same restore commands aimed at the live data directory with the server stopped, then starting it again.

Retention

The collection rows a job produces are what grows over time. The retention sweep prunes them on the schedule the project set:

automaton retention sweep -project acme -data /var/lib/automaton
automaton retention history -project acme -data /var/lib/automaton

Retention is a project setting rather than a deployment setting, so an organization with a records policy sets it per project and the sweep enforces it. What the sweep removes is the collection rows, not the receipts: a receipt is the record that the call happened, and the audit record is written to outlive the data a call touched.

Erasing on request

When a subject asks for their data to be destroyed, one collection at a time is destroyed by destroying the key that opens it:

automaton erase -data /var/lib/automaton -project acme -handle col_... -reason "subject request 4417"

The rows become unreadable. What survives is the accounting: how many rows there were, how large they were, when the erase happened, and the reason you gave. That is deliberate, and it is what lets you answer the next question the request generates, which is proof that the erase happened.