Gateway
Webhooks and Slack
Every merge, hold, conflict and undo sent to your backend, signed, or to a Slack channel in words with links to approve and undo.
Self-hostedChronos Gateway runs on your own servers today. Releases and install script. The engine under it: Chronos DB, self-hosted
What a webhook is for
Everything that happens to a run is an event: it was started, its merge was written, held, in conflict, rejected or undone, it expired, was discarded or failed. The gateway keeps every event in its store and sends each one to every webhook that takes it.
There are two formats. json sends the event itself, signed, to your backend or your log pipeline. slack sends the event as a message a person reads, to a Slack incoming webhook, with a link to approve a held merge and a link to undo a merged one.
Webhooks are the gateway’s, not one database’s: with several databases behind one gateway, one webhook hears about all of them, and every event’s data says which (source). This page describes gateway 0.3.3.
Register a webhook
POST /v1/webhooks with a developer key. The answer (201) holds the signing secret, shown only then. Keep it with your backend’s secrets.
curl -s "$GATEWAY/v1/webhooks" -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \ -d '{"url": "https://app.example.com/hooks/chronos", "events": ["merge.held", "merge.landed", "merge.conflict"]}'{"id": "wh_...", "url": "https://app.example.com/hooks/chronos", "events": ["merge.held", "merge.landed", "merge.conflict"], "format": "json", "created": 1767225600000, "secret": "whsec_..."}| Field | Required | Meaning |
|---|---|---|
| url | yes | Where deliveries go. It must start with http:// or https://. For slack, the URL of a Slack incoming webhook. |
| events | no | The event types to send, as an array. Left out or empty: every event. |
| format | no | json (the default): the event, signed. slack: a message for a channel. Anything else is refused (400). |
The console does the same under Webhooks: add one (its signing secret is shown once) or remove it.
List and delete
| Call | Answers |
|---|---|
| GET /v1/webhooks | {"webhooks": [{id, url, events, format, created}]}. The secret is never shown again. |
| DELETE /v1/webhooks/:id | {"ok": true}. 404 when there is no webhook with that id. |
There is no call to change a webhook or to show its secret again. To change the URL or the events, or to rotate the secret, register a new webhook and delete the old one.
A delivery
Each delivery is an HTTP POST with a JSON body and these headers:
| Header | Holds |
|---|---|
| Content-Type | application/json |
| Chronos-Event-Id | The event’s id. It is unique, so a receiver that gets one twice (a retry after a lost answer) can tell. |
| Chronos-Signature | t=<unix seconds>,v1=<hex HMAC-SHA256 of "<t>.<body>">, made with the webhook’s secret. |
The body of a json delivery is the event:
{ "id": "evt_...", "type": "merge.landed", "run_id": "run_...", "data": {"run": "run_...", "rows": 12, "by": "rules", "source": "default"}, "created": 1767225600000}| Field | Meaning |
|---|---|
| id | The event’s id, the same as the Chronos-Event-Id header. |
| type | The event type, from the table below. |
| run_id | The run it is about. Null for run.refused, where no run was made. |
| data | The event’s own fields. Always has source: the database the event is about (default for the first). |
| created | When it happened, in milliseconds since 1970. |
Answer with any 2xx status within 10 seconds. Anything else, a timeout or a redirect counts as failed, and the delivery is tried again. Do the work after you answer, or in a queue: a slow receiver is retried and gets the event twice.
Events
| Type | When | Data |
|---|---|---|
| run.created | A run was started | run, tenant, intent, pg_role (the role of act_as, when given; its settings are never shown); parent for a child run (next release) |
| merge.landed | A run’s changes were written to the database | run, rows, by (rules, person or reviewer; from 0.4.0 also parent, a child run merged into its parent’s world, and reconciled, a merge found in the engine’s history after a lost answer), approved_by (when a person or the reviewer approved), partial (a part was approved), reads_checked: false (approved with skip_read_check) |
| merge.held | A proposed run broke a rule and waits for a person | run, reasons, end_user, world, version, policy, asked, owner, changes; from 0.4.1 strict ({channels, until}) for a change only a person may approve, within its window: see Strict changes |
| merge.reviewed | The AI reviewer gave its verdict on a held run | run, world, version, verdict (approve, reject or pass), reason, confidence, by: {reviewer, model} |
| merge.conflict | A row the run changed was changed in the database too. Nothing was written | run, message |
| merge.rejected | A person rejected a held run | run, by, reason |
| merge.undone | A merged run was undone, wholly or in part | run, rows, skipped, by |
| merge.undo_conflict | An undo was refused: rows the run wrote have changed since | run, by |
| run.expired | A run’s time ran out, or a held run waited past GATEWAY_HOLD_TTL | run, was (its status before) |
| run.discarded | A run was discarded (DELETE /v1/runs/:id, or its database was disconnected) | run |
| run.failed | A run can’t go on: the engine failed its merge, or its copy of the data was lost with a crashed engine | run, and message; or code: "world_lost" and error |
| run.refused | A tenant went past GATEWAY_RUNS_PER_HOUR. No run was made | tenant, runs_in_the_last_hour, limit |
- A run the reviewer approves sends
merge.reviewedand thenmerge.landedwithby: "reviewer". One it rejects or passes stays held, for a person. - A decision made signed in records the person:
approved_by: {end_user, person: {email, name, sub}, via: "sso"}. One made in the console says"via": "console"; one made with a developer key says"via": "api"and thebyyour backend sent. - A merge, discard or undo that Chronos Cloud asked for (when linked) goes to your webhooks as one made through the API would.
Effects: what a merge sets off
A merge can set off something outside the database: a refund, an email. In a run’s world that is NOTIFY ON MERGE, and the engine keeps it in its outbox until a worker says it is done. Webhooks tell you about runs; effects are what the merged change itself asks for. With GATEWAY_EFFECTS set, the gateway is that worker.
GATEWAY_EFFECTS=billing=https://billing.internal/hook,*=https://all.internal/hookGATEWAY_EFFECTS_SECRET=<a long random secret>- A receiver per channel,
channel=URLpairs;*for any other channel. An effect whose channel has no receiver waits, without using its attempts. - Claimed, kept, then acknowledged. The gateway (its leader, with several) claims effects from the engine (
CLAIM OUTBOX ... IDEMPOTENT), keeps each in its store, and only then acknowledges it to the engine (ACK OUTBOX). - Delivered as a
POSTof{id, channel, payload, world, approved, source, attempt}, withIdempotency-Key: <the effect’s id>andChronos-Signature, signed withGATEWAY_EFFECTS_SECRETas webhooks are signed. Check it withverify_signatureorverifySignature. - Retried on anything but 2xx after 10 s, 1 min, 5 min, 30 min, 2 h and 6 h. After
GATEWAY_EFFECTS_MAX_ATTEMPTS(8) it is given up,dead, and kept with why. - At least once. A receiver may see an effect twice when an answer is lost: drop the repeat by its idempotency key.
GET /v1/effects(developer key;?status=pending,deliveredordead, andlimit) lists them newest first, each with its status, attempts, next try, last error and the receiver’s receipt.GATEWAY_EFFECTSneedsGATEWAY_EFFECTS_SECRET: the gateway won’t start with the first and not the second.- An undo does not unsend an effect. An undo’s preview lists them as
effects_not_undone.
Example bodies
The data of each event. The envelope around it is the same every time.
{"run": "run_...", "tenant": {"column": "org_id", "value": 42}, "intent": "remove the duplicate lead", "pg_role": null, "source": "default"}{"run": "run_...", "rows": 12, "by": "rules", "source": "default"}{"run": "run_...", "rows": 3, "by": "person", "approved_by": {"end_user": null, "person": {"email": "ada@acme.com", "name": "Ada", "sub": "00u1..."}, "via": "sso"}, "source": "default"}{"run": "run_...", "reasons": ["deletes 1 row, more than 0"], "end_user": "user_7", "world": "...", "version": 4, "policy": "gateway", "asked": "...", "owner": "...", "changes": [...], "source": "default"}{"run": "run_...", "world": "...", "version": 4, "verdict": "pass", "reason": "A refund over 200 needs a person.", "confidence": 0.62, "by": {"reviewer": "gateway_reviewer", "model": "qwen3:32b"}, "source": "default"}{"run": "run_...", "message": "...", "source": "default"}{"run": "run_...", "by": {"by": "ada@example.com", "via": "api"}, "reason": "still a customer", "source": "default"}{"run": "run_...", "rows": 11, "skipped": 1, "by": {"by": "ada@example.com", "via": "api"}, "source": "shop"}{"run": "run_...", "by": {"by": "ada@example.com", "via": "api"}, "source": "shop"}{"run": "run_...", "was": "held", "source": "default"}{"run": "run_...", "source": "default"}{"run": "run_...", "code": "world_lost", "error": "...", "source": "default"}{"tenant": {"column": "org_id", "value": 42}, "runs_in_the_last_hour": 60, "limit": 60, "source": "default"}The signature
Every delivery carries Chronos-Signature: t=<unix seconds>,v1=<signature>. The signature is the HMAC-SHA256, in hex, of the string <t>.<body>, keyed with the webhook’s secret: the timestamp, a dot, then the raw body exactly as it was received.
- Read the raw body before anything parses it. A body that was parsed and written again may not be byte for byte the same.
- Split the header at commas, then each part at the first
=. Taketand everyv1. - Refuse a
ttoo far from now. The client libraries allow 300 seconds. - Work out HMAC-SHA256 of
<t>.<body>with the secret, in hex. - Compare it with each
v1in constant time. One match is enough.
The client libraries do this: verify_signature in Python (chronos-gateway) and verifySignature in TypeScript (@chronosdb/gateway).
from chronos_gateway import verify_signature@app.post("/hooks/chronos")async def chronos_hook(request): body = await request.body() # the raw bytes header = request.headers["Chronos-Signature"] if not verify_signature(WEBHOOK_SECRET, header, body): # tolerance: 300 seconds return Response(status_code=400) event = json.loads(body) if already_seen(event["id"]): # a retry return Response(status_code=200) enqueue(event) # do the work after answering return Response(status_code=200)import hashlib, hmac, timedef verify(secret: str, header: str, body: bytes, tolerance: int = 300) -> bool: parts = [p.split("=", 1) for p in header.split(",") if "=" in p] try: t = int(next(v for k, v in parts if k.strip() == "t")) except (StopIteration, ValueError): return False if abs(time.time() - t) > tolerance: return False want = hmac.new(secret.encode(), f"{t}.{body.decode()}".encode(), hashlib.sha256).hexdigest() return any(hmac.compare_digest(v, want) for k, v in parts if k.strip() == "v1")import { createHmac, timingSafeEqual } from 'node:crypto';export function verify(secret, header, rawBody, tolerance = 300) { const parts = header.split(',').map((p) => p.trim().split(/=(.*)/s)); const t = Number(parts.find(([k]) => k === 't')?.[1]); if (!Number.isInteger(t) || Math.abs(Date.now() / 1000 - t) > tolerance) return false; const want = Buffer.from(createHmac('sha256', secret).update(t + '.' + rawBody).digest('hex')); return parts.some(([k, v]) => k === 'v1' && v.length === want.length && timingSafeEqual(Buffer.from(v), want));}Retries
A delivery that doesn’t get a 2xx answer is tried again on this schedule, then dropped:
| Try | After the one before |
|---|---|
| 1 | at once |
| 2 | 10 seconds |
| 3 | 1 minute |
| 4 | 5 minutes |
| 5 | 30 minutes |
| 6 | 2 hours |
| 7 | 6 hours |
That is seven tries over about 8 hours 36 minutes. After the last one the gateway logs gave up on event ... with the URL and the reason, and the delivery counts as failed. Each try is signed with its own timestamp. Redirects are not followed.
A retry can arrive after a later event, and an event can arrive twice. Use Chronos-Event-Id to drop repeats, and read the run (GET /v1/runs/:id) when order matters.
The backlog
GET /v1/status says how deliveries stand, across every webhook:
"webhooks": {"pending": 0, "failed": 2}| Count | Meaning |
|---|---|
| pending | Deliveries waiting for their first try or their next one. A number that keeps growing means a receiver is down or slow. |
| failed | Deliveries given up on after the last try. |
The console’s Overview shows the same. A failed delivery isn’t sent again by itself: read what was missed from GET /v1/events.
Past events
Every event is kept, delivered or not. GET /v1/events lists them with a developer key.
| Query | Answers |
|---|---|
| ?after=<event id>&limit= | Events after that one, oldest first. limit is 100 unless given, 1,000 at most. Without after: from the first. |
| ?last=N | The newest N (1,000 at most), oldest of them first. |
curl -s "$GATEWAY/v1/events?after=evt_...&limit=500" -H "Authorization: Bearer $KEY"{"events": [{"id": "evt_...", "type": "merge.held", "run_id": "run_...", "data": {...}, "created": 1767225600000}]}To follow events without a webhook, keep the id of the last event you handled and ask for those after it. This is also the way to send the audit trail to a SIEM: point a webhook at your log pipeline, or read the events on a timer.
Slack
Make an incoming webhook in Slack for the channel, and register its URL with "format": "slack":
curl -s "$GATEWAY/v1/webhooks" -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \ -d '{"url": "https://hooks.slack.com/services/T000/B000/XXXX", "format": "slack", "events": ["merge.held", "merge.landed", "merge.conflict", "merge.undone"]}'Each event is then sent as {"text": "..."}, a message, in place of the event’s JSON. events works the same: without it the channel gets every event, run.created included. The message is made once per delivery, from the event and its run. A retry sends the same message, with the same approval or undo link, not a new link each time. A gateway restart between retries makes one more.
Words that come from your data (a run’s intent, a reason) have &, < and > escaped, so nobody’s words make a link or a mention in the channel.
What each message says
Each message names the run by its intent, or by its id when it has none.
| Event | Message |
|---|---|
| merge.held | Waiting for approval: the intent, the number of changes, then Held: and the reasons. A second line links to review it. |
| merge.landed | Merged into Postgres: the intent, the number of rows, and how: within the rules, approved by a person, a part approved by a person or approved by the AI reviewer. A second line links to undo it. |
| merge.conflict | Conflict: the intent and the engine’s message. |
| merge.rejected | Rejected: the intent, and the reason when one was given. |
| merge.reviewed | AI reviewer approved, AI reviewer rejected, waiting for a person or AI reviewer passed it to a person, with the intent and the model’s reason. |
| merge.undone | Undone: the intent, the number of rows, and how many changed since were left as they are. |
| merge.undo_conflict | Undo refused: the intent, and that rows it wrote have changed since. |
| run.created | Run started: the intent. |
| run.expired | Expired: the intent. |
| run.discarded | Discarded: the intent. |
| run.failed | Failed: the intent. |
| run.refused | Run refused: the tenant is past its limit of runs an hour. |
:raised_hand: *Waiting for approval:* reassign open leads to Ada, 2 changes. Held: deletes 3 rows, more than 2<https://gateway.example.com/approve#cga_...|Review and approve or reject>:white_check_mark: *Merged into Postgres:* reassign open leads to Ada (12 rows), within the rules<https://gateway.example.com/approve#cgu_...|Review and undo>:leftwards_arrow_with_hook: *Undone:* reassign open leads to Ada (2 rows), 1 changed since left as it isThe database is named. A merge’s message says which database it was written to: Merged into Postgres, Merged into MySQL, Merged into MariaDB or Merged into MongoDB, from the database the run works beside. Before 0.3.2 a merge beside MariaDB said MySQL.
Approving from Slack
A held merge’s message carries a link, Review and approve or reject. The gateway makes an approval for it, as POST /v1/runs/:id/approvals makes one, and links to its page on the gateway (/approve#<token>). The token is after the #, so it never reaches a server’s log.
- The page shows the change in plain words, with Approve and Reject.
- The link works once, until
GATEWAY_APPROVAL_TTL(1 day by default), and approves only the version of the run it was made for. A run that changed since answersstale. - It is a bearer link: whoever is in the channel and opens it decides. To know who decided and to let only your approvers decide, turn on single sign-on (below).
- Approving writes the change with the same checks as any merge. A conflict or changed reads is said on the page.
- If the approval can’t be made, the message says
(no approval link: ...)with the reason.
People must be able to open the gateway’s address (GATEWAY_PUBLIC_URL) from their browser for the link to work.
Undoing from Slack
A merge’s message carries a link too, Review and undo: a page that shows what undoing that merge would put back, in words, and undoes it.
- The link holds a
cgu_token. It works once, untilGATEWAY_APPROVAL_TTL, for whoever holds it. - The undo is the same checked one as
POST /v1/runs/:id/undo: every row locked and compared, one transaction. - A row someone changed again since the merge is never overwritten. The page says which, and offers to undo the rest and leave those as they are.
- An undo that would change rows besides the run’s own (a cascade or a trigger in the database) is refused there. It is done from the console, by someone who has looked at which.
- The page shows changes in words only, never a row’s values.
- A run undone but for rows changed since (
partly_undone) can be undone again for those rows later, from the console or the API. See Undo. - The link is sent only for a run that merged and has an undo record. Undo itself is offered for 30 days after the merge, but the link ends with
GATEWAY_APPROVAL_TTL: after that, undo from the console or the API.
| Setting | Default | Meaning |
|---|---|---|
| GATEWAY_SLACK_UNDO | on | off (or 0, false, no): a merge’s message carries no undo link. |
| GATEWAY_APPROVAL_TTL | 1d | How long an approval link works, and an undo link in a Slack message. |
The page’s calls
The undo page uses the widget’s endpoint with the cgu_ token as the bearer. GET /v1/approve answers {kind: "undo", run, rows, changes, changed_since, cascades}, each change as {kind, text, table, key}. POST /v1/approve with {"decision": "undo"} undoes; add "skip_changed": true to leave rows changed since as they are.
| Answer | Code | Meaning |
|---|---|---|
| 401 | invalid | The link isn’t valid. |
| 409 | used | This was already undone. |
| 410 | expired | The link expired. |
| 409 | changed_since | Rows this change wrote have been changed again since. Nothing was undone. Undo the rest with skip_changed. |
| 409 | cascade | Undoing would change rows besides its own. Undo it from the console. |
| 409 | undo_not_ready | Undo isn’t set up for this database yet. Whoever runs the gateway sets it up from its console. |
With single sign-on
When single sign-on is set up (GATEWAY_OIDC_ISSUER), nothing in the channel approves or undoes anything by itself. Both links go to pages behind sign-in:
| Message | Without single sign-on | With single sign-on |
|---|---|---|
| A held merge | Review and approve or reject: an approval link, /approve#cga_... | Sign in to review and approve or reject: the change on /approvals, for a person on GATEWAY_OIDC_ALLOW |
| A merge | Review and undo: an undo link, /approve#cgu_... | Open it in the console to undo: the run in the console, for a person on GATEWAY_OIDC_ADMINS |
A decision made signed in records who made it (approved_by), in the merge.landed event and on the run. See Approvals for setting up single sign-on, and Undo for what an undo checks.
Other formats
The gateway described here has two formats, json and slack. A format it doesn’t have is refused (400): a webhook's format is "json" (the event, signed) or "slack". A binary built with extensions can add formats and event types; GET /v1/status lists what is built in under extensions, and the open gateway has none.
Operating notes
- The gateway calls out. The gateway sends nothing outside your network unless you set up webhooks or link it to Chronos Cloud. With the Helm chart’s NetworkPolicy on, allow port 443 to your webhook targets (
networkPolicy.https.cidrs). - One sender. With several gateways sharing a store, only the leader delivers, so an event isn’t sent once per gateway.
- Up to 50 deliveries are sent at a time, one after another. A receiver that takes the full 10 seconds to answer slows the others. Answer fast.
- Tracing. With
OTEL_EXPORTER_OTLP_ENDPOINTset, each delivery is a span namedwebhookwith the event’s id and the attempt. - Use HTTPS. An
http://URL is taken, and the event’s body then crosses the network in the clear. The signature shows who sent it; it doesn’t hide it. - Events can hold your data.
merge.heldcarrieschangesandreasons, which can quote rows. Send webhooks only where that may go.