Chronos/DBWaitlist

Gateway

Undo

Put back what a merged run wrote, each row checked first: from the API, the console or a link in Slack.

Self-hostedChronos Gateway runs on your own servers today. Releases and install script. The engine under it: Chronos DB, self-hosted

What undo is

Undo puts back every row a merged run wrote, as it was before the merge. A row the run changed gets its old values. A row the run removed is added again. A row the run added is removed.

It goes through the database as any merge goes: each row is locked and compared first, and all of it is written in one transaction. A row someone changed again since the merge is never overwritten. Such a row either refuses the undo or, if you say so, stays as it is while the rest goes back.

  • A run can be undone for 30 days after its merge. You can make that longer: there is no upper limit. See How long.
  • Undo works beside Postgres, MySQL, MariaDB and MongoDB.
  • A big run takes seconds, not minutes, from 0.3.3: undoing 10,000 rows took 2.7 seconds beside Postgres on a laptop. With engine 0.5.2 or older the same undo took about a minute, as each row’s check read the whole table.
  • Three ways in: POST /v1/runs/:id/undo with a developer key, the Undo button on the run’s page in the console, and a link in a Slack message of the merge.
  • A run that was approved in parts is undone whole: every merge of it goes back.
  • A run undone but for rows changed since can be undone again for those rows. See Undo again.

This page describes gateway 0.4.4 on Chronos DB 0.6.2.

What it needs

  • A run with merges to put back. A merged run; a partly_undone run, for the rows left; or a run that ended rejected, expired or discarded after a part of it was approved, for what that part wrote.
  • Engine 0.5.0 or later. From it, the engine gives each merge an id, and the gateway keeps the ids with the run (merges).
  • The record table in your database. Undo keeps what each merge wrote in a table of the database itself. See The record table.
  • A developer key, the console, or an undo link.

GET /v1/status says whether the database is ready, under attached.undo.ready.

Preview first

With "preview": true nothing is written. The answer says what would go back, in words, and which rows changed since.

POST /v1/runs/:id/undo, a preview
curl -s localhost:7400/v1/runs/$RUN/undo -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \  -d '{"preview": true}'
the answer
{  "preview": true,  "status": "merged",  "rows": 2,  "skipped": 1,  "changes": [    {"kind": "update", "text": "Change the owner of lead Acme from Ada to Bo", "table": "leads", "key": "leads/2",     "before": {...}, "after": {...}, "status": "put_back"},    {"kind": "insert", "text": "Add lead Hooli", "table": "leads", "key": "leads/3",     "before": {...}, "after": {...}, "status": "put_back"}  ],  "changed_since": [    {"kind": "delete", "text": "Remove note 7", "table": "notes", "key": "notes/7",     "status": "changed", "now": {...}}  ],  "cascades": [],  "effects_not_undone": [{"id": "...", "channel": "...", "state": "..."}]}
FieldMeaning
previewTrue: nothing was written.
statusThe run’s status: merged, or partly_undone, rejected, expired or discarded for a run undone again or approved in part.
rowsHow many rows would go back.
skippedHow many rows changed since the merge and would be left.
changesThe rows that would go back. Each is the undo of a row, as a change: before is how the merge left the row, after is how it was before the merge. text says it in words: “Change the owner of lead Acme from Ada to Bo”, “Remove note 7”, and “Add lead Hooli” for a row brought back.
changed_sinceThe rows changed again since the merge. Each has the row as it is now.
cascadesBeside Postgres: the tables the database would change rows of besides the run’s own. Null beside MySQL, which cannot say. See Cascades.
effects_not_undoneEmails and webhooks the merge queued, as {id, channel, state}. They are never unsent.
  • Words follow your labels, as approvals do. See Rules, labels and masks.
  • Values of masked columns read (masked), in before, after and now.
  • Beside MongoDB a document’s change reads by field path: “Change the address.city of lead “Acme” from Pune to Mumbai”.
  • A preview is a dry run. Rows can still change between the preview and the undo, and the undo checks again.

Undo

Without preview, the undo is done.

POST /v1/runs/:id/undo
curl -s localhost:7400/v1/runs/$RUN/undo -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \  -d '{"by": "ada@example.com"}'# {"status": "undone", "rows": 3, "skipped": 0, "changed_since": [], "effects_not_undone": []}
FieldTakesMeaning
previewtrue or falseTrue: write nothing, and say what would go back.
skip_changedtrue or falseTrue: leave the rows changed since the merge as they are, and put the rest back. False, the default: such rows refuse the whole undo.
allow_cascadetrue or falseTrue: go on even though the database would change rows besides the run’s own. False, the default: such an undo is refused.
byany JSONWho undid it, as your backend knows them. Kept in the run’s outcome and sent in merge.undone. From the console, the person signed in is recorded instead.
Answer fieldMeaning
statusundone when every row went back; partly_undone when some were left.
rowsHow many rows this undo put back.
skippedHow many rows were left because they changed since.
changed_sinceThe rows left, each in words with the row as it is now.
effects_not_undoneWhat the merge queued that stays sent.

A flag that is not true or false answers 400. The body may be empty.

Rows changed since

Between the merge and the undo, your app, a person or another run may have changed a row the run wrote. The engine compares every row with the database before it writes, so it sees this.

  • Without it: with no skip_changed, the undo is refused with 409 changed_since. Nothing is written. merge.undo_conflict is sent.
  • With it: with "skip_changed": true, those rows stay as they are, theirs, and the rest go back. The run becomes partly_undone.
  • Preview first to see which rows: they are in changed_since, each with the row now.
undo the rest
curl -s localhost:7400/v1/runs/$RUN/undo -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \  -d '{"skip_changed": true, "by": "ada@example.com"}'# {"status": "partly_undone", "rows": 2, "skipped": 1, "changed_since": [...], "effects_not_undone": []}

Undo again

New in 0.3.2. A partly_undone run can be undone again, the same way, for the rows that were left. They go back once they hold what the run wrote.

  • How a row gets there. It is changed back to what the run wrote: through a new run, or directly in the database. A direct change is seen when GATEWAY_FOLLOW is on, or after a refresh.
  • Rows the first undo put back are not counted. They are as they were before the run already, so they are not “changed since” and never refuse the second undo.
  • A row still changed is treated as in the first undo: it refuses the undo with 409 changed_since, or with "skip_changed": true it stays as it is and the run stays partly_undone.
  • The counts. rows in the answer is what this undo put back. outcome.undone.rows on the run is the total over both undos.
  • The status. The run becomes undone when no row is left.
  • With an engine older than 0.5.2, the rows the first undo put back refuse the second undo unless it says "skip_changed": true.
  • In the console the button on a partly_undone run reads “Undo the rest”.
undo again, once the row left holds what the run wrote
curl -s localhost:7400/v1/runs/$RUN/undo -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \  -d '{"by": "ada@example.com"}'# {"status": "undone", "rows": 1, "skipped": 0, "changed_since": [], "effects_not_undone": []}# GET /v1/runs/$RUN then has outcome.undone.rows: 3

After a part was approved

A held run can have a part approved, with only_tables or only_keys. That part merges at once, and the rest stays held. If the run then ends rejected, expired or discarded, the part’s rows are still in the database.

  • New in 0.3.2: such a run can be undone for what that part wrote, with POST /v1/runs/:id/undo or from the console.
  • The console shows a note on such a run, that a part of it was approved and merged before it ended, and an Undo button.
  • The undo is the same checked one. The run becomes undone, or partly_undone when rows changed since were left.
  • While the run is still held, approve or reject the rest first. A run that is still live is not undone.

How long: the undo window

A run can be undone for as long as its record is kept. That is 30 days after the merge by default.

  • Change it with CHRONOS_UNDO_WINDOW_DAYS, a whole number of days, in the gateway’s environment. The gateway hands it to the engine it starts. With an engine you run yourself, set it there.
  • There is no upper limit. 90, 365 or 3650 all work. The number must be more than 0; anything else means 30.
  • What it costs is space in your database. Every merge keeps the rows it wrote, before and after, in the record table (chronos.merge_rows; chronos_merge_rows beside MySQL, MariaDB and MongoDB). A longer window keeps them longer.
  • Old records go as new merges are written. After each merge the engine deletes records past the window, a few thousand at a time. A role that may not delete them leaves them for an administrator.
  • A longer window does not bring records back. It covers records still there when you set it, and every merge after. A record already deleted under a shorter window is gone: the undo of its run is refused, and the message says the merge is older than the undo window.
  • A shorter window deletes sooner. Records older than the new window go at the next merges.
  • Read what is set: GET /v1/status answers attached.undo.window_days.
a year of undo, with the image
docker run -d --name chronos-gateway -p 127.0.0.1:7400:7400 -v chronos-gateway:/data \  -e CHRONOS_UNDO_WINDOW_DAYS=365 \  ghcr.io/abhishekxdg/chronos-gateway:0.4.4

Cascades

Undoing can make the database change rows the run did not write. The usual case is ON DELETE CASCADE: the run added a row, something else later added rows under it, and removing the run’s row would remove those too. A trigger can do the same.

  • A preview names the tables in cascades, beside Postgres. Beside MySQL and MariaDB it is null: the database cannot say.
  • An undo that would cascade is refused with 409 cascade, until the request says "allow_cascade": true.
  • The console asks before it sends that. An undo link never sends it.

Effects are not undone

A merge can queue effects: emails, webhooks, payments sent once it lands. Undo puts rows back. It does not unsend anything.

  • A preview and an undo both list them in effects_not_undone, as {id, channel, state}.
  • The policy rule effects_delay makes effects wait a set time after a merge lands. An undo within that time takes them back unsent. See Rules, labels and masks.
  • The gateway’s own webhooks about the merge (merge.landed) were already delivered. Undo sends merge.undone after them.

Statuses and the record

StatusMeaning
undoneMerged, then put back: every row it wrote is as it was before. Final: it is not undone again.
partly_undoneMerged, then put back but for the rows changed since, which stay as they are. It can be undone again for those rows. See Undo again.

The run keeps its merge’s outcome, and gains who undid it and what went back. After a second undo, undone.rows counts every row put back over both, and undone_by is who did the latest:

GET /v1/runs/:id, outcome
{"merged": 3, "by": "rules", "undone_by": {"by": "ada@example.com", "via": "api"}, "undone": {"rows": 2, "skipped": 1, "merge": "...", "effects_not_undone": []}}
Undone throughundone_by
The API, with a developer key{"by": <what your backend sent as by>, "via": "api"}
The console{"by": <email or name>, "person": {...}, "via": "console"}
An undo link{"via": "undo_link"}

Both statuses can be listed: GET /v1/runs?status=undone and ?status=partly_undone. The console’s Runs page has both as filters.

Errors

CodeStatusMeaning and what to do
changed_since409Rows the run wrote have changed since. Nothing was written, and merge.undo_conflict is sent. Preview the undo to see them, then undo with skip_changed to leave them as they are. With an engine older than 0.5.2, a partly_undone run answers this on a second undo until it says skip_changed.
cascade409The database would change rows besides the run’s own. Preview the undo to see which tables, then undo with allow_cascade.
undo_not_ready409The gateway’s role has no table to keep what merges write. The error carries sql: the statements a database administrator runs once. Runs merged after that can be undone.
no_undo_record409The run’s merge has no undo record: it merged before the gateway kept merge ids, or the engine is older than 0.5.0.
open, held, conflict, failed409The run is still live: approve, reject or discard it first. The code is the run’s status.
undone409The run was already undone whole. Nothing is left to put back.
rejected, expired, discarded409The run ended without any merge, so it wrote nothing to undo. One that had a part approved is undone and does not answer this.
no_source404The run’s database was disconnected from the gateway.
409 undo_not_ready
{"error": {"code": "undo_not_ready",  "message": "undo isn't set up for this database: the gateway's role has no table to keep what merges write. A database administrator runs this once, and runs merged after that can be undone:\n...",  "sql": "create table if not exists chronos.merge_rows (...);\n..."}}

A 400 means a flag was not true or false. A 401 means no developer key was sent. Any other refusal is the engine’s, with its own code and message.

The record table

Each merge writes what it changed into a table of your database, beside the table of merges. Undo reads it. The connection check’s role_sql makes it and grants it to the gateway’s role.

DatabaseTableThe role’s rights on it
Postgreschronos.merge_rowsselect, insert, delete
MySQL, MariaDBchronos_merge_rows, in the attached databaseselect, insert, delete
MongoDBthe collection chronos_merge_rowsfind, insert, remove, createCollection, createIndex

A role made from the statements of gateway 0.3.0 or earlier has no such table. Merges were never affected, but undo was not ready there. From 0.3.1 the gateway says so plainly: GET /v1/status answers attached.undo.ready: false, and an undo answers 409 undo_not_ready with the statements below in error.sql. A database administrator runs them once. Runs merged after that can be undone; runs merged before it cannot.

Postgres
create table if not exists chronos.merge_rows (merge_id text not null, n int not null, table_name text not null, op text not null, key jsonb, before jsonb, after jsonb, at timestamptz not null default now(), primary key (merge_id, n));create index if not exists merge_rows_at on chronos.merge_rows (at);grant select, insert, delete on chronos.merge_rows to chronos_gateway;
MySQL and MariaDB
create table if not exists chronos_merge_rows (merge_id varchar(255) not null, n int not null, table_name varchar(255) not null, op varchar(10) not null, row_key json, before_row json, after_row json, at datetime(6) not null default current_timestamp(6), primary key (merge_id, n), key chronos_merge_rows_at (at)) engine = InnoDB;grant select, insert, delete on chronos_merge_rows to 'chronos_gateway'@'%';
MongoDB (mongosh)
db.grantPrivilegesToRole("chronos_gateway", [{ resource: { db: db.getName(), collection: "chronos_merge_rows" }, actions: ["find", "insert", "remove", "createCollection", "createIndex"] }])
  • The statements name the role chronos_gateway. If your role has another name, use it.
  • The table holds, per row a merge wrote: the merge’s id, the table, the operation, the row’s key, and the row before and after. So it holds values from the attached tables. It lives in your database, under your database’s own access rules.
  • The gateway’s role may read, add and delete its rows, nothing more.

In the console

A merged run’s page, under Runs, has an Undo button. It always shows the preview first, then one button. On a partly_undone run the button reads “Undo the rest”.

  • The preview lists what goes back in words, says how many rows were changed again by someone else and that the undo leaves them alone, names the tables a cascade would touch, and lists what is not undone: “What was already sent stays sent.”
  • The button reads “Undo” when every row can go back, “Undo the rest” when some changed since, and “Undo, cascade too” when the database would change other rows.
  • Nothing to undo is shown when every row the run wrote has been changed again since.
  • Not set up: on a database without the record table, the console shows the statements to run, with a Copy button.
  • Undone in part before: when the rows left are as they were before the run already, put back by hand since, there is nothing to write. The console marks the run undone and says every row is as it was before the run.
  • A part approved: a run that ended rejected, expired or discarded after a part of it was approved has a note saying that part was merged, and an Undo button that puts those rows back.
  • No button: a run that merged before undo records were kept says it cannot be undone here.
  • Afterwards the run’s page says it was merged, then undone, by whom, how many rows were put back and how many were left as changed since.
  • The person signed in to the console is recorded as who undid it.

The Undo link in Slack

A webhook with format: "slack" posts each merge to a channel: “Merged into Postgres” (or MySQL, MariaDB or MongoDB), the intent, the rows, and how it was approved. From 0.3.1 the message carries a link, Review and undo.

  • The token. The link is GATEWAY_PUBLIC_URL + /approve#cgu_.... The cgu_ token is after the #, so browsers never send it to a server and it is in no log. The gateway keeps only its hash.
  • Once. The link undoes that one merged run, one time, for whoever holds it. Anyone in the channel can use it.
  • One link per message. The message is made once per delivery, so a retry carries the same link. A gateway restart between retries makes one more.
  • For a time. It works until GATEWAY_APPROVAL_TTL, 1 day by default, counted from the message.
  • Off. GATEWAY_SLACK_UNDO=off sends no undo link. It is on by default.
  • With single sign-on, the message links to the run in the console instead, behind sign-in: “Open it in the console to undo”. Nothing in the channel undoes anything by itself.

What the page shows

  • The heading “Undo this change?”, the run’s intent, and how many rows go back, “each checked first”.
  • What goes back, each row in words.
  • The rows changed again by someone else since, which stay as they are.
  • What is not undone: the effects already sent.
  • One button: “Undo”, or “Undo the rest” when some rows changed since. That sends skip_changed.
  • “Nothing to undo” when every row the run wrote has been changed again since.

What a link cannot do

  • Never a cascade. A link never sends allow_cascade. When the undo would change rows besides the run’s own, the page says so, the button is off, and the undo is done from the console by someone who has looked at which.
  • Words, not row values. A link’s holder gets each change as {kind, text, table, key}: what it is, in words, and where. Never before, after or now. Whole rows are for a developer key and the console.
  • Never over a newer edit. It is the same checked undo as the API’s. A row changed again since is never overwritten.
  • No set-up. On a database without the record table, the page says undo is not set up yet and that whoever runs the gateway can set it up from its console. The statements are not shown.

The page’s calls

The page is the approval widget with an undo token. Its calls are the widget’s, with the cgu_ token as the bearer token.

CallBodyAnswers
GET /v1/approve{kind: "undo", run: {id, intent, tenant, created}, expires, rows, changes, changed_since, cascades, effects_not_undone}
POST /v1/approve{"decision": "undo"}, with "skip_changed": true to undo the rest{status, rows, skipped, changed_since, effects_not_undone}
CodeStatusMeaning
invalid401The link is not one the gateway knows.
used409This was already undone with this link.
expired410The link’s time ran out. Undo from the console or the API.
changed_since409Rows the change wrote were changed again since. Nothing was undone. The page shows them and offers to undo the rest.
cascade409Undoing would change rows besides its own. It is undone from the console.
undo_not_ready409Undo is not set up for this database yet.
undone409The change was already undone whole, another way.

An undo that was not carried out frees the link: after changed_since, for example, the same link can undo the rest. A link that has undone the rest is used: the rows left are undone later from the console or the API. A link not yet used still works on a run that was partly_undone another way. A decision other than undo answers 400.

Webhooks

EventdataWhen
merge.undone{run, rows, skipped, by}An undo was done, whole or in part. skipped counts the rows left as changed since.
merge.undo_conflict{run, by}An undo was refused because rows changed since, and nobody said to leave them. Nothing was written.

A Slack webhook says both in a line: “Undone”, the intent, the rows put back and how many changed since were left; and “Undo refused: rows it wrote have changed since”. With several databases, every event’s data says source. See Webhooks and Slack.

With the gateway linked to Chronos Cloud, the cloud may ask for an undo only if GATEWAY_CLOUD_ALLOW has undo, which it has by default.

What undo does not cover

  • Effects already sent. Emails, webhooks and payments stay sent. They are listed, not unsent.
  • Rows changed since. They stay as they are. Undo never overwrites a newer edit.
  • Runs merged longer ago than the undo window: 30 days unless you set it. See How long.
  • Runs merged before undo records were kept: with an engine older than 0.5.0, or before the record table was made and granted in your database.
  • Runs that are still live: open, held, conflict or failed. A held run that was approved in part has merged rows, but it is undone only once the run is merged or has ended.
  • A run already undone, and a run that ended without any merge.
  • Rows of a partly undone run that are still changed. They go back only once they hold what the run wrote.
  • A cascade from a link. That is done from the console or the API, with allow_cascade.

Next

Undo · Chronos