Chronos/DBWaitlist

Gateway

Webhooks and Slack

Every merge, hold, conflict and undo sent to your backend, signed, or to a Slack channel in words with links to approve and undo.

Self-hostedChronos Gateway runs on your own servers today. Releases and install script. The engine under it: Chronos DB, self-hosted

What a webhook is for

Everything that happens to a run is an event: it was started, its merge was written, held, in conflict, rejected or undone, it expired, was discarded or failed. The gateway keeps every event in its store and sends each one to every webhook that takes it.

There are two formats. json sends the event itself, signed, to your backend or your log pipeline. slack sends the event as a message a person reads, to a Slack incoming webhook, with a link to approve a held merge and a link to undo a merged one.

Webhooks are the gateway’s, not one database’s: with several databases behind one gateway, one webhook hears about all of them, and every event’s data says which (source). This page describes gateway 0.3.3.

Register a webhook

POST /v1/webhooks with a developer key. The answer (201) holds the signing secret, shown only then. Keep it with your backend’s secrets.

register
curl -s "$GATEWAY/v1/webhooks" -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \  -d '{"url": "https://app.example.com/hooks/chronos", "events": ["merge.held", "merge.landed", "merge.conflict"]}'{"id": "wh_...", "url": "https://app.example.com/hooks/chronos", "events": ["merge.held", "merge.landed", "merge.conflict"], "format": "json", "created": 1767225600000, "secret": "whsec_..."}
FieldRequiredMeaning
urlyesWhere deliveries go. It must start with http:// or https://. For slack, the URL of a Slack incoming webhook.
eventsnoThe event types to send, as an array. Left out or empty: every event.
formatnojson (the default): the event, signed. slack: a message for a channel. Anything else is refused (400).

The console does the same under Webhooks: add one (its signing secret is shown once) or remove it.

List and delete

CallAnswers
GET /v1/webhooks{"webhooks": [{id, url, events, format, created}]}. The secret is never shown again.
DELETE /v1/webhooks/:id{"ok": true}. 404 when there is no webhook with that id.

There is no call to change a webhook or to show its secret again. To change the URL or the events, or to rotate the secret, register a new webhook and delete the old one.

A delivery

Each delivery is an HTTP POST with a JSON body and these headers:

HeaderHolds
Content-Typeapplication/json
Chronos-Event-IdThe event’s id. It is unique, so a receiver that gets one twice (a retry after a lost answer) can tell.
Chronos-Signaturet=<unix seconds>,v1=<hex HMAC-SHA256 of "<t>.<body>">, made with the webhook’s secret.

The body of a json delivery is the event:

an event
{  "id": "evt_...",  "type": "merge.landed",  "run_id": "run_...",  "data": {"run": "run_...", "rows": 12, "by": "rules", "source": "default"},  "created": 1767225600000}
FieldMeaning
idThe event’s id, the same as the Chronos-Event-Id header.
typeThe event type, from the table below.
run_idThe run it is about. Null for run.refused, where no run was made.
dataThe event’s own fields. Always has source: the database the event is about (default for the first).
createdWhen it happened, in milliseconds since 1970.

Answer with any 2xx status within 10 seconds. Anything else, a timeout or a redirect counts as failed, and the delivery is tried again. Do the work after you answer, or in a queue: a slow receiver is retried and gets the event twice.

Events

TypeWhenData
run.createdA run was startedrun, tenant, intent, pg_role (the role of act_as, when given; its settings are never shown); parent for a child run (next release)
merge.landedA run’s changes were written to the databaserun, rows, by (rules, person or reviewer; from 0.4.0 also parent, a child run merged into its parent’s world, and reconciled, a merge found in the engine’s history after a lost answer), approved_by (when a person or the reviewer approved), partial (a part was approved), reads_checked: false (approved with skip_read_check)
merge.heldA proposed run broke a rule and waits for a personrun, reasons, end_user, world, version, policy, asked, owner, changes; from 0.4.1 strict ({channels, until}) for a change only a person may approve, within its window: see Strict changes
merge.reviewedThe AI reviewer gave its verdict on a held runrun, world, version, verdict (approve, reject or pass), reason, confidence, by: {reviewer, model}
merge.conflictA row the run changed was changed in the database too. Nothing was writtenrun, message
merge.rejectedA person rejected a held runrun, by, reason
merge.undoneA merged run was undone, wholly or in partrun, rows, skipped, by
merge.undo_conflictAn undo was refused: rows the run wrote have changed sincerun, by
run.expiredA run’s time ran out, or a held run waited past GATEWAY_HOLD_TTLrun, was (its status before)
run.discardedA run was discarded (DELETE /v1/runs/:id, or its database was disconnected)run
run.failedA run can’t go on: the engine failed its merge, or its copy of the data was lost with a crashed enginerun, and message; or code: "world_lost" and error
run.refusedA tenant went past GATEWAY_RUNS_PER_HOUR. No run was madetenant, runs_in_the_last_hour, limit
  • A run the reviewer approves sends merge.reviewed and then merge.landed with by: "reviewer". One it rejects or passes stays held, for a person.
  • A decision made signed in records the person: approved_by: {end_user, person: {email, name, sub}, via: "sso"}. One made in the console says "via": "console"; one made with a developer key says "via": "api" and the by your backend sent.
  • A merge, discard or undo that Chronos Cloud asked for (when linked) goes to your webhooks as one made through the API would.

Effects: what a merge sets off

A merge can set off something outside the database: a refund, an email. In a run’s world that is NOTIFY ON MERGE, and the engine keeps it in its outbox until a worker says it is done. Webhooks tell you about runs; effects are what the merged change itself asks for. With GATEWAY_EFFECTS set, the gateway is that worker.

deliver effects
GATEWAY_EFFECTS=billing=https://billing.internal/hook,*=https://all.internal/hookGATEWAY_EFFECTS_SECRET=<a long random secret>
  • A receiver per channel, channel=URL pairs; * for any other channel. An effect whose channel has no receiver waits, without using its attempts.
  • Claimed, kept, then acknowledged. The gateway (its leader, with several) claims effects from the engine (CLAIM OUTBOX ... IDEMPOTENT), keeps each in its store, and only then acknowledges it to the engine (ACK OUTBOX).
  • Delivered as a POST of {id, channel, payload, world, approved, source, attempt}, with Idempotency-Key: <the effect’s id> and Chronos-Signature, signed with GATEWAY_EFFECTS_SECRET as webhooks are signed. Check it with verify_signature or verifySignature.
  • Retried on anything but 2xx after 10 s, 1 min, 5 min, 30 min, 2 h and 6 h. After GATEWAY_EFFECTS_MAX_ATTEMPTS (8) it is given up, dead, and kept with why.
  • At least once. A receiver may see an effect twice when an answer is lost: drop the repeat by its idempotency key.
  • GET /v1/effects (developer key; ?status= pending, delivered or dead, and limit) lists them newest first, each with its status, attempts, next try, last error and the receiver’s receipt.
  • GATEWAY_EFFECTS needs GATEWAY_EFFECTS_SECRET: the gateway won’t start with the first and not the second.
  • An undo does not unsend an effect. An undo’s preview lists them as effects_not_undone.

Example bodies

The data of each event. The envelope around it is the same every time.

run.created
{"run": "run_...", "tenant": {"column": "org_id", "value": 42}, "intent": "remove the duplicate lead", "pg_role": null, "source": "default"}
merge.landed, within the rules
{"run": "run_...", "rows": 12, "by": "rules", "source": "default"}
merge.landed, approved by a person signed in
{"run": "run_...", "rows": 3, "by": "person", "approved_by": {"end_user": null, "person": {"email": "ada@acme.com", "name": "Ada", "sub": "00u1..."}, "via": "sso"}, "source": "default"}
merge.held
{"run": "run_...", "reasons": ["deletes 1 row, more than 0"], "end_user": "user_7", "world": "...", "version": 4, "policy": "gateway", "asked": "...", "owner": "...", "changes": [...], "source": "default"}
merge.reviewed
{"run": "run_...", "world": "...", "version": 4, "verdict": "pass", "reason": "A refund over 200 needs a person.", "confidence": 0.62, "by": {"reviewer": "gateway_reviewer", "model": "qwen3:32b"}, "source": "default"}
merge.conflict
{"run": "run_...", "message": "...", "source": "default"}
merge.rejected
{"run": "run_...", "by": {"by": "ada@example.com", "via": "api"}, "reason": "still a customer", "source": "default"}
merge.undone
{"run": "run_...", "rows": 11, "skipped": 1, "by": {"by": "ada@example.com", "via": "api"}, "source": "shop"}
merge.undo_conflict
{"run": "run_...", "by": {"by": "ada@example.com", "via": "api"}, "source": "shop"}
run.expired
{"run": "run_...", "was": "held", "source": "default"}
run.discarded
{"run": "run_...", "source": "default"}
run.failed, the run’s copy lost
{"run": "run_...", "code": "world_lost", "error": "...", "source": "default"}
run.refused
{"tenant": {"column": "org_id", "value": 42}, "runs_in_the_last_hour": 60, "limit": 60, "source": "default"}

The signature

Every delivery carries Chronos-Signature: t=<unix seconds>,v1=<signature>. The signature is the HMAC-SHA256, in hex, of the string <t>.<body>, keyed with the webhook’s secret: the timestamp, a dot, then the raw body exactly as it was received.

  1. Read the raw body before anything parses it. A body that was parsed and written again may not be byte for byte the same.
  2. Split the header at commas, then each part at the first =. Take t and every v1.
  3. Refuse a t too far from now. The client libraries allow 300 seconds.
  4. Work out HMAC-SHA256 of <t>.<body> with the secret, in hex.
  5. Compare it with each v1 in constant time. One match is enough.

The client libraries do this: verify_signature in Python (chronos-gateway) and verifySignature in TypeScript (@chronosdb/gateway).

Python, with the client library
from chronos_gateway import verify_signature@app.post("/hooks/chronos")async def chronos_hook(request):    body = await request.body()                       # the raw bytes    header = request.headers["Chronos-Signature"]    if not verify_signature(WEBHOOK_SECRET, header, body):   # tolerance: 300 seconds        return Response(status_code=400)    event = json.loads(body)    if already_seen(event["id"]):                     # a retry        return Response(status_code=200)    enqueue(event)                                    # do the work after answering    return Response(status_code=200)
Python, without it
import hashlib, hmac, timedef verify(secret: str, header: str, body: bytes, tolerance: int = 300) -> bool:    parts = [p.split("=", 1) for p in header.split(",") if "=" in p]    try:        t = int(next(v for k, v in parts if k.strip() == "t"))    except (StopIteration, ValueError):        return False    if abs(time.time() - t) > tolerance:        return False    want = hmac.new(secret.encode(), f"{t}.{body.decode()}".encode(), hashlib.sha256).hexdigest()    return any(hmac.compare_digest(v, want) for k, v in parts if k.strip() == "v1")
Node, without it
import { createHmac, timingSafeEqual } from 'node:crypto';export function verify(secret, header, rawBody, tolerance = 300) {  const parts = header.split(',').map((p) => p.trim().split(/=(.*)/s));  const t = Number(parts.find(([k]) => k === 't')?.[1]);  if (!Number.isInteger(t) || Math.abs(Date.now() / 1000 - t) > tolerance) return false;  const want = Buffer.from(createHmac('sha256', secret).update(t + '.' + rawBody).digest('hex'));  return parts.some(([k, v]) => k === 'v1' && v.length === want.length && timingSafeEqual(Buffer.from(v), want));}

Retries

A delivery that doesn’t get a 2xx answer is tried again on this schedule, then dropped:

TryAfter the one before
1at once
210 seconds
31 minute
45 minutes
530 minutes
62 hours
76 hours

That is seven tries over about 8 hours 36 minutes. After the last one the gateway logs gave up on event ... with the URL and the reason, and the delivery counts as failed. Each try is signed with its own timestamp. Redirects are not followed.

A retry can arrive after a later event, and an event can arrive twice. Use Chronos-Event-Id to drop repeats, and read the run (GET /v1/runs/:id) when order matters.

The backlog

GET /v1/status says how deliveries stand, across every webhook:

in GET /v1/status
"webhooks": {"pending": 0, "failed": 2}
CountMeaning
pendingDeliveries waiting for their first try or their next one. A number that keeps growing means a receiver is down or slow.
failedDeliveries given up on after the last try.

The console’s Overview shows the same. A failed delivery isn’t sent again by itself: read what was missed from GET /v1/events.

Past events

Every event is kept, delivered or not. GET /v1/events lists them with a developer key.

QueryAnswers
?after=<event id>&limit=Events after that one, oldest first. limit is 100 unless given, 1,000 at most. Without after: from the first.
?last=NThe newest N (1,000 at most), oldest of them first.
read what happened since the last event you handled
curl -s "$GATEWAY/v1/events?after=evt_...&limit=500" -H "Authorization: Bearer $KEY"{"events": [{"id": "evt_...", "type": "merge.held", "run_id": "run_...", "data": {...}, "created": 1767225600000}]}

To follow events without a webhook, keep the id of the last event you handled and ask for those after it. This is also the way to send the audit trail to a SIEM: point a webhook at your log pipeline, or read the events on a timer.

Slack

Make an incoming webhook in Slack for the channel, and register its URL with "format": "slack":

a Slack webhook
curl -s "$GATEWAY/v1/webhooks" -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \  -d '{"url": "https://hooks.slack.com/services/T000/B000/XXXX", "format": "slack",       "events": ["merge.held", "merge.landed", "merge.conflict", "merge.undone"]}'

Each event is then sent as {"text": "..."}, a message, in place of the event’s JSON. events works the same: without it the channel gets every event, run.created included. The message is made once per delivery, from the event and its run. A retry sends the same message, with the same approval or undo link, not a new link each time. A gateway restart between retries makes one more.

Words that come from your data (a run’s intent, a reason) have &, < and > escaped, so nobody’s words make a link or a mention in the channel.

What each message says

Each message names the run by its intent, or by its id when it has none.

EventMessage
merge.heldWaiting for approval: the intent, the number of changes, then Held: and the reasons. A second line links to review it.
merge.landedMerged into Postgres: the intent, the number of rows, and how: within the rules, approved by a person, a part approved by a person or approved by the AI reviewer. A second line links to undo it.
merge.conflictConflict: the intent and the engine’s message.
merge.rejectedRejected: the intent, and the reason when one was given.
merge.reviewedAI reviewer approved, AI reviewer rejected, waiting for a person or AI reviewer passed it to a person, with the intent and the model’s reason.
merge.undoneUndone: the intent, the number of rows, and how many changed since were left as they are.
merge.undo_conflictUndo refused: the intent, and that rows it wrote have changed since.
run.createdRun started: the intent.
run.expiredExpired: the intent.
run.discardedDiscarded: the intent.
run.failedFailed: the intent.
run.refusedRun refused: the tenant is past its limit of runs an hour.
messages, as text
:raised_hand: *Waiting for approval:* reassign open leads to Ada, 2 changes. Held: deletes 3 rows, more than 2<https://gateway.example.com/approve#cga_...|Review and approve or reject>:white_check_mark: *Merged into Postgres:* reassign open leads to Ada (12 rows), within the rules<https://gateway.example.com/approve#cgu_...|Review and undo>:leftwards_arrow_with_hook: *Undone:* reassign open leads to Ada (2 rows), 1 changed since left as it is

The database is named. A merge’s message says which database it was written to: Merged into Postgres, Merged into MySQL, Merged into MariaDB or Merged into MongoDB, from the database the run works beside. Before 0.3.2 a merge beside MariaDB said MySQL.

Approving from Slack

A held merge’s message carries a link, Review and approve or reject. The gateway makes an approval for it, as POST /v1/runs/:id/approvals makes one, and links to its page on the gateway (/approve#<token>). The token is after the #, so it never reaches a server’s log.

  • The page shows the change in plain words, with Approve and Reject.
  • The link works once, until GATEWAY_APPROVAL_TTL (1 day by default), and approves only the version of the run it was made for. A run that changed since answers stale.
  • It is a bearer link: whoever is in the channel and opens it decides. To know who decided and to let only your approvers decide, turn on single sign-on (below).
  • Approving writes the change with the same checks as any merge. A conflict or changed reads is said on the page.
  • If the approval can’t be made, the message says (no approval link: ...) with the reason.

People must be able to open the gateway’s address (GATEWAY_PUBLIC_URL) from their browser for the link to work.

Undoing from Slack

A merge’s message carries a link too, Review and undo: a page that shows what undoing that merge would put back, in words, and undoes it.

  • The link holds a cgu_ token. It works once, until GATEWAY_APPROVAL_TTL, for whoever holds it.
  • The undo is the same checked one as POST /v1/runs/:id/undo: every row locked and compared, one transaction.
  • A row someone changed again since the merge is never overwritten. The page says which, and offers to undo the rest and leave those as they are.
  • An undo that would change rows besides the run’s own (a cascade or a trigger in the database) is refused there. It is done from the console, by someone who has looked at which.
  • The page shows changes in words only, never a row’s values.
  • A run undone but for rows changed since (partly_undone) can be undone again for those rows later, from the console or the API. See Undo.
  • The link is sent only for a run that merged and has an undo record. Undo itself is offered for 30 days after the merge, but the link ends with GATEWAY_APPROVAL_TTL: after that, undo from the console or the API.
SettingDefaultMeaning
GATEWAY_SLACK_UNDOonoff (or 0, false, no): a merge’s message carries no undo link.
GATEWAY_APPROVAL_TTL1dHow long an approval link works, and an undo link in a Slack message.

The page’s calls

The undo page uses the widget’s endpoint with the cgu_ token as the bearer. GET /v1/approve answers {kind: "undo", run, rows, changes, changed_since, cascades}, each change as {kind, text, table, key}. POST /v1/approve with {"decision": "undo"} undoes; add "skip_changed": true to leave rows changed since as they are.

AnswerCodeMeaning
401invalidThe link isn’t valid.
409usedThis was already undone.
410expiredThe link expired.
409changed_sinceRows this change wrote have been changed again since. Nothing was undone. Undo the rest with skip_changed.
409cascadeUndoing would change rows besides its own. Undo it from the console.
409undo_not_readyUndo isn’t set up for this database yet. Whoever runs the gateway sets it up from its console.

With single sign-on

When single sign-on is set up (GATEWAY_OIDC_ISSUER), nothing in the channel approves or undoes anything by itself. Both links go to pages behind sign-in:

MessageWithout single sign-onWith single sign-on
A held mergeReview and approve or reject: an approval link, /approve#cga_...Sign in to review and approve or reject: the change on /approvals, for a person on GATEWAY_OIDC_ALLOW
A mergeReview and undo: an undo link, /approve#cgu_...Open it in the console to undo: the run in the console, for a person on GATEWAY_OIDC_ADMINS

A decision made signed in records who made it (approved_by), in the merge.landed event and on the run. See Approvals for setting up single sign-on, and Undo for what an undo checks.

Other formats

The gateway described here has two formats, json and slack. A format it doesn’t have is refused (400): a webhook's format is "json" (the event, signed) or "slack". A binary built with extensions can add formats and event types; GET /v1/status lists what is built in under extensions, and the open gateway has none.

Operating notes

  • The gateway calls out. The gateway sends nothing outside your network unless you set up webhooks or link it to Chronos Cloud. With the Helm chart’s NetworkPolicy on, allow port 443 to your webhook targets (networkPolicy.https.cidrs).
  • One sender. With several gateways sharing a store, only the leader delivers, so an event isn’t sent once per gateway.
  • Up to 50 deliveries are sent at a time, one after another. A receiver that takes the full 10 seconds to answer slows the others. Answer fast.
  • Tracing. With OTEL_EXPORTER_OTLP_ENDPOINT set, each delivery is a span named webhook with the event’s id and the attempt.
  • Use HTTPS. An http:// URL is taken, and the event’s body then crosses the network in the clear. The signature shows who sent it; it doesn’t hide it.
  • Events can hold your data. merge.held carries changes and reasons, which can quote rows. Send webhooks only where that may go.
Webhooks and Slack · Chronos