Chronos/DBWaitlist

Gateway

Configure and operate

Every setting, tokens and what each can do, high availability, backups, upgrades and what to check when something is off.

Self-hostedChronos Gateway runs on your own servers today. Releases and install script. The engine under it: Chronos DB, self-hosted

This page is the reference for running the gateway: gateway 0.4.4 on Chronos DB (the engine) 0.6.2, image ghcr.io/abhishekxdg/chronos-gateway. Installing it is in Install and set up.

How settings are read

The gateway reads its settings from environment variables, once, when it starts. A setting that is wrong stops it with a sentence saying what it takes.

InstallWhere settings goApply them
Installer, --service/etc/chronos-gateway/gateway.envsudo systemctl restart chronos-gateway
Docker-e NAME=value on docker runRemove the container and run it again
Docker Compose.env beside compose.yamldocker compose up -d
HelmThe chart’s values; env for any GATEWAY_* settinghelm upgrade
  • Times take 60, 60s, 15m, 1h or 1d. A bare number is seconds.
  • For limits, 0 means no limit.
  • Switches take on or off (also 1, true, yes and 0, false, no).
  • chronos-gateway --help lists the settings too.

Where it runs

VariableDefaultMeaning
GATEWAY_LISTEN127.0.0.1:7400 (image: 0.0.0.0:7400)Address and port it listens on.
GATEWAY_PUBLIC_URLhttps:// + GATEWAY_DOMAIN, else http:// + GATEWAY_LISTEN (http://localhost:7400 when it listens on all addresses)How clients and browsers reach it. Used in sql_url, mcp_url, approval links and the console, which only answers pages opened at this address. Use https:// anywhere but this machine.
GATEWAY_DOMAINunsetIts domain behind HTTPS (gateway.example.com, no https://, no path): sets GATEWAY_PUBLIC_URL to https://<domain> when that is unset. Compose’s https profile gets its certificate.
GATEWAY_DATAgateway-data (service: /var/lib/chronos-gateway, image: /data)Its data folder.
GATEWAY_ALLOWED_ORIGINSnoneWeb origins allowed to call the approval endpoints from a browser, comma-separated (https://app.example.com). The sites that show the approval widget.
GATEWAY_API_KEYunsetOne more developer key to accept, besides those made with keys create. It can’t be revoked with keys revoke: remove it and restart.

The engine and your database

VariableDefaultMeaning
CHRONOS_BINthe engine the installer put beside the gateway (<prefix>/lib/chronos-gateway/chronos), else chronos on the PATHThe engine binary the gateway starts.
CHRONOS_ENGINE_URL, CHRONOS_ADMIN_TOKENunsetUse an engine you run yourself instead of starting one. Both are needed. Then attach where that engine runs, and call POST /v1/setup. One database only, and not with several gateways.
GATEWAY_REFRESH_EVERY5mHow often the engine takes your database’s latest rows. Each refresh reads the attached tables in full.
GATEWAY_FOLLOWoffon: the engine also takes the database’s changes about a second after they commit. See below.
CHRONOS_UNDO_WINDOW_DAYS30How many days after its merge a run can be undone. No upper limit; a longer window keeps more rows in the record table in your database. See Undo.
GATEWAY_SOURCE_FILEunsetA file holding the database URL (a mounted secret). Copied into the data folder at each start, so a changed password is picked up at the next restart. While it is set, POST /v1/connect takes tables only and refuses a url.
GATEWAY_TABLESunsetWith GATEWAY_SOURCE_FILE: the tables to attach at the first start, comma-separated.
GATEWAY_TENANT_COLUMNunsetWith GATEWAY_TABLES: the tenant column.
GATEWAY_FIELDSunsetWith GATEWAY_SOURCE_FILE, beside MongoDB: the fields kept as columns, comma-separated.

GATEWAY_FOLLOW, per database

BesideHow it followsNeeds
PostgresA logical replication slotwal_level = logical and a role with REPLICATION. The connect check’s follow says what is missing.
MongoDBA change stream on each attached collectionEngine 0.5.1 or later, and changeStream on those collections for the gateway’s role. Nothing to set up or clean up.
MySQL, MariaDBNot yetThe gateway refreshes on its timer.

The start-up settings (GATEWAY_SOURCE_FILE, GATEWAY_TABLES and the like) are for the first database only. One connected under a name is connected through the API or the console.

Rules and limits

VariableDefaultMeaning
GATEWAY_POLICYgatewayName of the merge policy runs keep to.
GATEWAY_POLICY_RULESmax_rows = 500, max_deletes = 0, rows_per_hour = 5000, max_age = '15 minutes', check_reads = true, require_scope = trueThe rules a change must keep to be written without a person. It replaces the whole list. Rules set in the console or with PUT /v1/policy are kept in the store and win over it.
GATEWAY_RUN_TTL1hHow long a run, and its token, lasts. A run can ask for less (ttl_seconds), not more.
GATEWAY_HOLD_TTL3dHow long a held change waits for a person before it expires.
GATEWAY_APPROVAL_TTL1dHow long an approval link works, at most. An undo link in a Slack message too.
GATEWAY_SLACK_UNDOonA Slack message of a merge carries a link that undoes it (once, for whoever holds it, rows changed since left alone). off: no link.
GATEWAY_RUN_STATEMENTS_PER_MINUTE120SQL statements, MCP tool calls and MongoDB tool calls per run per minute. Past it: 429.
GATEWAY_RUN_PROPOSES_PER_MINUTE6Proposes per run per minute. Each one locks and compares rows in your database. Past it: 429 too_many_proposes.
GATEWAY_RUNS_PER_HOUR60New runs per tenant per hour. Past it: 429 too_many_runs, and a run.refused event.
GATEWAY_RUN_MAX_ROWS1000Rows one SQL answer returns to an agent. Longer answers are cut and marked truncated.

The default rules, in plain words: a change is written without a person only if it deletes no rows, changes at most 500 rows, keeps its tenant under 5,000 changed rows an hour (counted across all its runs), was made within the last 15 minutes, every row it read still matches the database, and it declares its tenant (every gateway run does). Anything else is held.

The rules are applied when you connect. To change them with the variable later, set it, restart, then apply it:

apply GATEWAY_POLICY_RULES again
curl -s -X POST "$GATEWAY/v1/setup" -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d '{}'

Effect delivery

From 0.4.0. With these set, the gateway delivers what merges queue in the engine’s outbox (NOTIFY ON MERGE). How it works is in Effects.

VariableDefaultMeaning
GATEWAY_EFFECTSnone: effects are not deliveredchannel=URL pairs, comma separated, * for any other channel: billing=https://billing.internal/hook,*=https://all.internal/hook. Channel names are letters, digits, _, - and .; receivers are http(s) URLs.
GATEWAY_EFFECTS_SECRETnoneSigns each delivery (Chronos-Signature). Required with GATEWAY_EFFECTS: the gateway won’t start without it.
GATEWAY_EFFECTS_MAX_ATTEMPTS8Tries before an effect is given up as dead. At least 1.

Single sign-on

Unset by default: approval links work by themselves. Set these so people sign in with your OpenID Connect provider to approve and to use the console.

VariableDefaultMeaning
GATEWAY_OIDC_ISSUERunsetYour provider’s issuer URL (https://acme.okta.com, https://login.microsoftonline.com/<tenant>/v2.0, https://accounts.google.com). https:// only, or plain http:// to this machine. Register <GATEWAY_PUBLIC_URL>/sso/callback as the redirect URI.
GATEWAY_OIDC_CLIENT_ID, GATEWAY_OIDC_CLIENT_SECRETunsetThe gateway’s client at your provider. Both are needed. The secret is sent with HTTP Basic (client_secret_basic).
GATEWAY_OIDC_ALLOWunset, requiredWho may approve: emails (ada@acme.com), domains (@acme.com) and groups (group:db-approvers), comma-separated. Without it the gateway doesn’t start.
GATEWAY_OIDC_ADMINSunsetWho may sign in to the console and change settings, in the same form. They may also approve.
GATEWAY_OIDC_GROUPS_CLAIMgroupsThe ID token’s claim that lists a person’s groups (roles with Entra ID app roles).
GATEWAY_OIDC_REQUIREoffon: every approval link needs a person on the allow list, signed in. Off: only links made for an approver.
GATEWAY_OIDC_SESSION_TTL12hHow long a sign-in lasts.

AI reviewer

A model decides each held change before a person does. Unset: people decide them all.

VariableDefaultMeaning
GATEWAY_REVIEWER_URLunsetAn OpenAI-compatible API’s base: https://api.openai.com/v1, http://localhost:11434/v1 (Ollama), http://vllm:8000/v1. Plain http only to this machine, or without a key.
GATEWAY_REVIEWER_MODELunset, requiredThe model to ask. Any model works, hosted or open-weight.
GATEWAY_REVIEWER_KEYunsetIts API key, sent as a bearer token.
GATEWAY_REVIEWER_MODEL_2unsetA second model on the same API. Both must approve, at the confidence below, for a change to merge.
GATEWAY_REVIEWER_MIN_CONFIDENCE0.8From 0 to 1. An approval less sure than this goes to a person.
GATEWAY_REVIEWER_INSTRUCTIONSunsetYour rules for its decisions, in words.
GATEWAY_REVIEWER_INSTRUCTIONS_FILEunsetA file to read those instructions from. It wins over the variable.
GATEWAY_REVIEWER_MAX_ROWS100The most changed rows it is shown.
GATEWAY_REVIEWER_COLUMNSunsetColumns whose text the model reads as it is: leads.name,leads.stage, leads.*, or *. Other text it sees only as a placeholder.
GATEWAY_REVIEWER_AGENTgateway_reviewerThe engine agent it decides as: letters, digits and _.

The reviewer works for the first database only. What it is shown and what it can’t approve are in Rules, labels and masks.

Link to Chronos Cloud

Optional, and unset by default: then nothing below runs. With GATEWAY_CLOUD set, the gateway opens one outbound TLS connection to Chronos Cloud so its dashboard can show reviews and send back what people decide there. The database URL and the engine’s admin token never cross that link. There is no inbound port.

VariableDefaultMeaning
GATEWAY_CLOUDunsetThe cloud’s https:// address.
GATEWAY_CLOUD_ENROLLunsetThe one-time enrollment token (cce_...) from the dashboard, for the first start only.
GATEWAY_CLOUD_ALLOWread,merge,discard,undoWhat the cloud may ask for. Also agents, policy, sql (read-only). Without merge, merging stays local.
GATEWAY_CLOUD_SHAREdiffsdiffs: changed rows, less the columns your labels hide; masked columns read masked. counts: no row leaves your network. Either way a cloud diff never returns SQL and a refusal loses its detail (from 0.4.3), and a cloud merge of a run’s world is an approval: of a run held or in conflict, at the version the person saw, into main.
HTTPS_PROXY, NO_PROXYunsetA proxy for that link (HTTP CONNECT), and hosts that skip it.

Several gateways

VariableDefaultMeaning
GATEWAY_STORE_URLunsetA Postgres database for the gateway’s own tables, shared by every gateway. Unset: the store is SQLite in the data folder, and there is one gateway.
GATEWAY_NODE_URLunsetWith GATEWAY_STORE_URL, required: how the other gateways reach this one (http://10.0.0.5:7400).
GATEWAY_SECRET_FILE<data>/secret.keyWith GATEWAY_STORE_URL: the file of the key that seals the store’s secrets. The same for every gateway sharing the store.

See High availability for what these give and need.

Logs and tracing settings

VariableDefaultMeaning
RUST_LOGchronos_gateway=infoLog detail, for example chronos_gateway=debug.
NO_COLORunsetSet: no colour codes in the log. Under systemd and Docker (no terminal) there are none anyway.
OTEL_EXPORTER_OTLP_ENDPOINTunsetAn OpenTelemetry collector, or a vendor’s OTLP/HTTP endpoint. Unset, nothing is sent.
OTEL_EXPORTER_OTLP_HEADERSunsetHeaders for it, such as a vendor’s key.
OTEL_SERVICE_NAMEchronos-gatewayThe service’s name in traces. OTEL_RESOURCE_ATTRIBUTES and the other standard OTEL_* variables work as usual.

With tracing on, the gateway sends a span for each request, named by its route (POST /v1/runs/:id/approve) with its status, a span for each call to the engine inside it, and one for each webhook delivery. A request with a W3C traceparent header joins the caller’s trace.

The command line

CommandDoes
chronos-gatewayRuns the gateway, the same as serve. Settings come from the environment.
chronos-gateway serveRuns the gateway. Ctrl-C or SIGTERM stops it: the engines are stopped cleanly first.
chronos-gateway keys create <name>Makes a developer key and prints it (cgk_...), once. Its id goes to standard error.
chronos-gateway keys listOne line per key: id, name, active or revoked.
chronos-gateway keys revoke <id>Revokes a key. A console sign-in made with it ends too.
chronos-gateway setup-codeA new code to set the gateway up in the browser, while it has no key yet. It works once, for a week.
chronos-gateway --helpThe usage and the settings. Also -h and help.
chronos-gateway --versionThe version. Also -V.

keys and setup-code open the store, so run them with the same GATEWAY_DATA (and GATEWAY_STORE_URL, if set) and as the same user as the gateway:

InstallMake a key
Installer, --servicesudo -u chronos-gateway env GATEWAY_DATA=/var/lib/chronos-gateway /usr/local/bin/chronos-gateway keys create backend
Dockerdocker exec chronos-gateway chronos-gateway keys create backend
Docker Composedocker compose exec gateway chronos-gateway keys create backend
Helmkubectl -n chronos exec statefulset/chronos-gateway -- chronos-gateway keys create backend

An unknown command prints the usage and exits with 2. A command that fails prints chronos-gateway: ... and exits with 1.

Files in the data folder

The data folder (GATEWAY_DATA) is readable by the gateway’s user only; the service makes it mode 700.

FileHolds
gateway.dbThe SQLite store: token hashes, runs, approvals, events, and webhook signing secrets (sealed from 0.4.4, with secret.key; those made before are kept as they are). Mode 600. Not there with GATEWAY_STORE_URL.
secret.keyThe key that seals the engine tokens of run agents and webhook secrets inside the store. Mode 600.
source.urlThe database URL, with its password. Mode 600.
engine.jsonThe engine’s local port and admin token. Mode 600.
db/The engine’s folder: the copy of the attached tables and the runs’ worlds.
engine.logThe engine’s output.
sources/<name>/The same files for a database connected under a name.
cloud.key, cloud.crt, cloud-ca.crt, cloud.json, cloud.hmacOnly when linked to Chronos Cloud: the connector’s key (600), its certificate, the cloud’s CA, the link’s address, and the key that hashes row ids under counts (600).

Tokens

Each token is 32 random bytes with a prefix. The gateway stores only their SHA-256 hashes.

TokenWho holds itWhat it opensHow long
cgk_ developer keyyour backendEverything except a run’s own SQL and MCP, and the widget’s calls. It can start runs and approve held changes.Until revoked (keys revoke <id>, the console, or DELETE /v1/keys/:id).
cgr_ run tokenone agentThat one run: its SQL, MCP, MongoDB tools, changes, status and propose.Until the run ends or its time runs out (GATEWAY_RUN_TTL).
cga_ approval tokenone personOne held change, at the version it was made for, once.GATEWAY_APPROVAL_TTL.
cgu_ undo linkwhoever is in the Slack channelUndo one merged change, once, as far as its rows are unchanged.GATEWAY_APPROVAL_TTL.
setup codewhoever reads the logMakes the first key, in the console. Works only while no key exists.Once, for 7 days; stops after 20 wrong tries.
sign-in cookiea person’s browserThe console (a key, or single sign-on on GATEWAY_OIDC_ADMINS), or approving (GATEWAY_OIDC_ALLOW).GATEWAY_OIDC_SESSION_TTL (12 hours). A key’s sign-in ends when the key is revoked.
  • Keep the developer key in your backend only. It never reaches your database directly, but it can start runs and approve held ones.
  • Make one key per backend, so each can be revoked alone.
  • The agent never holds a credential to your database. Behind a run token is an engine agent held to one tenant, whose own token works for the run’s time only.
  • The sign-in cookie is HttpOnly, SameSite=Lax, and Secure over https. It is kept as its hash. A person is checked against the allow list on every request, so removing someone from GATEWAY_OIDC_ALLOW and restarting ends their access.

What reaches your database

WhenWhat happens in your database
POST /v1/connect/checkThe gateway logs in with the URL you sent and reads the catalog. No writes. The URL isn’t kept.
POST /v1/connectThe engine logs in as the gateway’s role and reads the attached tables to copy them. The copy holds one read open while it runs.
Every GATEWAY_REFRESH_EVERYThe engine reads the attached tables again.
Every second, with GATEWAY_FOLLOWThe engine reads the changes since the last second from its replication slot (MongoDB: a change stream on each attached collection), then the rows they name.
An agent’s SQL, MCP or MongoDB tool callNothing. It runs in the run’s world.
A merge (rules kept, or a person approved)One transaction: each row the merge writes is locked and compared with the database, then written, and the merge is recorded in chronos.merges. If any row changed meanwhile, nothing is written.
An undoOne transaction of the same kind: each row locked and compared, then put back as it was.
Held, rejected, expired or discardedNothing.
  • The engine’s own tables are chronos.merges and chronos.merge_rows beside Postgres, chronos_merges and chronos_merge_rows beside MySQL and MariaDB, and collections of those names beside MongoDB. merge_rows keeps what each merge wrote, for undo.
  • Your app’s reads and writes never pass through the gateway.
  • With act_as, a run’s writes are made after SET LOCAL ROLE, as your end user’s role (Postgres only).
  • Between refreshes an agent may read a row that is out of date. That is safe: the merge compares every row it writes, and check_reads = true holds a change if a row it read has changed.

On the network

  • The engine listens on 127.0.0.1 only.
  • The gateway listens on 127.0.0.1:7400 unless you change it, over plain HTTP. Tokens travel in headers, so put TLS in front (nginx, Caddy, your load balancer) before anything on another machine calls it, and set GATEWAY_PUBLIC_URL to the https:// address.
  • The gateway logs a warning when it listens beyond loopback with an http:// public URL. In the image that is expected: it is fine while the port is published to 127.0.0.1 only.
  • Outbound, the gateway reaches your database, and only what you set up: webhook targets, your sign-on provider, an AI reviewer’s API, an OpenTelemetry collector, Chronos Cloud.
  • The Helm chart’s networkPolicy.enabled (off by default) lets the pod take calls on 7400 from its own namespace and reach only your database, DNS, and port 443 on the addresses you list.

High availability

One gateway on one data folder is the default, kept available by restarting it fast on the same data: the service unit restarts it on failure, and in containers one replica runs with its data on a volume that can move to another node. Since 0.3.0 several gateways can also run behind one address. One leads at a time; when it stops or is lost, another takes over.

What it gives, and what it doesn’t

  • The API stays up through the loss of a gateway or its node. A request may go to any gateway.
  • After a leader is lost, another leads within about 15 seconds (measured: 13 s after kill -9, 4 s after a clean stop). Requests in those seconds answer 503 no_leader with Retry-After: 5.
  • Merges pause for those seconds. A merge that was in flight is settled by the engine from the database’s own record of merges, as after any crash: it either happened whole or not at all.
  • A run that was open when the leader died hard loses its work. See Durability. A clean handover (a rolling update, SIGTERM) keeps open runs.
  • It does not make the database agents work on highly available: that is yours.

What it needs

  1. A store in Postgres, shared by every gateway: GATEWAY_STORE_URL=postgres://.... A database for the gateway’s own tables (keys, runs, approvals, events, the lease). It can live on the server agents work on, in a database of its own.
  2. One data folder, mounted by every gateway at GATEWAY_DATA: a ReadWriteMany volume (EFS, Filestore, Azure Files, CephFS, NFS). It holds the engines’ folders, the database URLs and the key that seals the store’s secrets. Only the leader’s engines write it.
  3. A node URL: GATEWAY_NODE_URL on each gateway: how the others reach it (http://10.0.0.5:7400). The leader’s goes into its lease, and followers send requests there. Allow that traffic between the gateways.
  4. The engine run by the gateway itself, not CHRONOS_ENGINE_URL.

If the gateways don’t share a data folder’s secret.key, give each the same key file with GATEWAY_SECRET_FILE. A gateway with GATEWAY_STORE_URL and no GATEWAY_NODE_URL doesn’t start.

How it works

PartHow
The leaseA row in the store (leases), taken in one statement, renewed every 5 seconds, good for 15. Time is the store’s clock, not each machine’s.
The leaderRuns the engines and the work that must happen once: the sweeper, webhook deliveries, the model reviewer, Chronos Cloud’s link.
A followerAnswers /health itself and sends every other request to the leader, unchanged, and returns the leader’s answer. The engines stay on the leader’s loopback.
Taking overA leader that stops gives the lease up. One that dies stops renewing; after 15 seconds a follower takes the lease and starts the engines on the shared folder.
Two engines never share a folderA leader that can’t renew for 8 seconds (the store unreachable, or the lease taken) stops its engines, kills any still there 4 seconds later, and exits: all before its 15-second lease runs out. Whatever runs it starts it again, as a follower. Measured with the store frozen: engine gone and gateway exited after 6.5 s. On Linux an engine is also told to stop when its gateway dies.
TimingValue
Lease renewedevery 5 seconds
Lease good for15 seconds
A leader gives up when it can’t renew for8 seconds
Engines still running are killed after4 more seconds
Retry-After on 503 no_leader5 seconds
A follower’s request to the leaderconnects within 3 seconds, waits up to 150 seconds
A statement the store doesn’t answerfails its request after 30 seconds

GET /v1/status says ha: {shared, node, role, leader: {node, url, expires}}. Point your load balancer’s health check at GET /health, which every gateway answers itself.

Kubernetes

values.yaml
ha:  enabled: true  replicas: 2  storeExistingSecret: chronos-store      # key storeUrl  existingClaim: chronos-gateway-shared   # ReadWriteMany

The chart sets GATEWAY_NODE_URL from each pod’s address. With a NetworkPolicy, allow the gateways’ pods to reach each other on 7400.

Limits

  • One connection to the store’s Postgres per gateway, one statement at a time.
  • A follower holds a request’s body in memory to send it on (64 MB at most).
  • The fence between an old leader and a new one is time (the lease) and, on Linux, the engine stopping with its gateway. A leader paused whole for longer than 15 seconds and then resumed (a frozen VM) runs for a fraction of a second more before its watch notices.
  • Not built: taking the engine’s folder over from object storage (so no shared volume is needed), a pool of store connections, moving an existing SQLite store into Postgres.

Durability

WhatThrough a clean stopThrough a hard crash
A merge already writtenKept: it is in your database.Kept.
A merge in flightThe gateway stops the engine cleanly before it exits.Settled when the engine starts, from the database’s record of merges (chronos.merges). Nothing is written twice and nothing is lost.
Runs, holds, approval links, events, keysKept in the store.Kept in the store.
An open run’s unmerged workKept.Not kept.

The engine keeps a world’s unmerged changes through a clean stop, not through a crash. After a hard crash (a kill -9, a lost node, a power cut) the next call on a run whose world was lost answers 409 world_lost. The run becomes failed, run.failed is sent with code: "world_lost", and nothing of it has reached the database. Start a new run. This holds for one gateway as much as for several.

So build the agent’s side to be able to start again: a run is one task, and a task that fails is run again from the top. GATEWAY_RUN_TTL (1 hour) already bounds how long any run’s work lives unmerged.

Backups

Back up the data folder along with your database. It holds the store, the keys and the engine’s folder. For a consistent copy, stop the gateway first:

back up the service’s data
sudo systemctl stop chronos-gatewaysudo tar -C /var/lib -czf chronos-gateway-backup.tar.gz chronos-gatewaysudo systemctl start chronos-gateway
  • With GATEWAY_STORE_URL, the store is in Postgres: back that database up as you back up any other.
  • The backup holds source.url and secret.key. Keep it as carefully as the database password.
  • Your data is in your database, not here. A lost data folder loses the gateway’s keys, runs waiting for approval, events and settings, not rows your app owns.
  • Agents can’t start runs or merge while the gateway is down. Your database and your app aren’t affected: the gateway is never in their path.

Upgrades and versions

Back up the data folder first. Upgrades keep settings, data and keys.

InstallUpgrade
Installer, --serviceRun the install command again. It replaces both binaries and restarts the service.
Installer, binaries onlyRun the install command again, then restart your chronos-gateway.
Dockerdocker pull ghcr.io/abhishekxdg/chronos-gateway:<version>, then docker rm -f chronos-gateway and run the same docker run command with the new tag. The volume carries over.
Docker ComposeSet the new GATEWAY_VERSION in .env, then docker compose pull && docker compose up -d.
Helmhelm upgrade chronos-gateway <the new release’s chart URL> -n chronos --reuse-values. The volume carries over.

Check the result with chronos-gateway --version. The bundled engine answers --version too:

InstallEngine version
Installer, --service/usr/local/lib/chronos-gateway/chronos --version
Installer, binaries only~/.local/lib/chronos-gateway/chronos --version
Dockerdocker exec chronos-gateway chronos --version
Helmkubectl -n chronos exec statefulset/chronos-gateway -- chronos --version

The engine a feature needs

Each gateway release bundles the engine it was tested with: gateway 0.4.4, 0.4.3, 0.4.2 and 0.4.1 with engine 0.6.2, gateway 0.4.0 with engine 0.6.0, gateway 0.3.4 with engine 0.5.5, gateway 0.3.3 with engine 0.5.3, gateway 0.3.2 with engine 0.5.2, gateway 0.3.1 with engine 0.5.1, gateway 0.3.0 with engine 0.5.0. The installer and the image install both together, so this matters only if you set CHRONOS_BIN or CHRONOS_ENGINE_URL to an engine of your own.

FeatureEngine
Masks, tables with row-level security on0.4.4 or later
Runs change child tables (reached through a foreign key)0.4.5
A run is held only by its own tenant’s rows0.4.6
act_as: runs write as your end user0.4.9
The reviewer’s policy rules (trusted_columns, watch_totals, canary, ...)0.4.10
A run’s reads checked against MySQL and MariaDB0.4.11 or later. With an older engine the default rules leave check_reads off; after upgrading it, POST /v1/setup turns the check on.
Rows added to AUTO_INCREMENT tables0.4.12 or later
Undo, MongoDB0.5.0 or later (MongoDB: an engine built with it)
GATEWAY_FOLLOW beside MongoDB0.5.1 or later
Evidence attached to the run’s world (in_engine: true), from gateway 0.4.0engine 0.6.0 (ATTACH EVIDENCE). With an older engine the evidence is kept with the run only
  • A role made from the statements of 0.3.0 or earlier has no table for undo. Merges work; GET /v1/status says attached.undo.ready: false, and an undo answers 409 undo_not_ready with the statements to run once.
  • A MongoDB role made from older statements has no changeStream. GATEWAY_FOLLOW then keeps refreshing on its timer until it is granted.
  • The 0.2.11 image can’t start its engine. Use 0.3.0 or later.
  • Each binary and the image are signed with cosign (since 0.2.6). CHRONOS_VERIFY=1 makes the installer check both.

Health and logs

CheckHow
Is it upGET /health answers {"ok": true} without a token. For load balancers and probes.
Is the engine healthyGET /v1/status with a key: engine.healthy.
Are webhooks getting throughGET /v1/status: webhooks.pending and webhooks.failed.
Who leadsGET /v1/status: ha.
The gateway’s logService: journalctl -u chronos-gateway. Docker: docker logs chronos-gateway. Compose: docker compose logs gateway.
The engine’s logengine.log in the data folder (/var/lib/chronos-gateway/engine.log; in Docker, docker cp chronos-gateway:/data/engine.log .).
An audit trailEvery run, hold, approval, merge and rejection is an event. Point a webhook at your log pipeline, or read GET /v1/events.
  • The gateway starts the engine again if it stops, and stops it cleanly on shutdown.
  • The service unit runs as chronos-gateway with a hardened sandbox (ProtectSystem=strict, NoNewPrivileges, write access to the data folder only), and restarts on failure.
  • The image runs as user 65532 on a distroless base, with no shell. Compose and the Helm chart add a read-only root filesystem and no Linux capabilities.
  • On big tables, refresh less often (GATEWAY_REFRESH_EVERY), or turn on GATEWAY_FOLLOW.

Troubleshooting

What you seeWhy, and what to do
401 with “send a developer key”The Authorization: Bearer cgk_... header is missing, or the token is a run or approval token.
/v1/connect/check: “not a Postgres URL: invalid value for option sslmode” or “unknown option”The check takes sslmode disable, prefer or require, and no other URL options. Drop verify-full, sslrootcert and the like for the check only. /v1/connect takes them.
/v1/connect/check: a certificate errorOnly sslmode=verify-ca or verify-full checks the certificate, against the public web CAs. With a private CA or a self-signed certificate, use sslmode=require for the check.
/v1/connect: “about N rows in all ... Send yes: true to go on” (400 confirm)Over 1,000,000 rows, attaching asks first: the copy holds one read open on the database while it runs (2.5 million rows took about 30 s on a test machine). Send the same request with "yes": true.
/v1/connect: “the server asks for the password as MD5” (or “in the clear”)Set password_encryption = 'scram-sha-256' and set the role’s password again, or connect with sslmode=verify-full.
/v1/connect: “no CA certificates found”verify-full found no system CA bundle. Add sslrootcert=<file.pem> with your CA, readable by the gateway’s user (in Docker, mount it).
/v1/connect in Docker can’t reach a database on the same machineInside the container, localhost is the container. Use host.docker.internal (on Linux add --add-host=host.docker.internal:host-gateway).
/v1/connect: 409 attachedThis gateway has a database already. Connect another under a name, or use a new data folder.
“the engine didn’t start within 30 s”Read engine.log in the data folder.
503 no_engineNo database is connected yet: POST /v1/connect.
503 no_leaderWith several gateways: none leads right now. Try again after Retry-After. If it lasts, check that the gateways reach the store and each other (GATEWAY_NODE_URL).
409 world_lost on a runThe engine crashed and the run’s unmerged work went with it. Start a new run.
POST /v1/runs: 400 tenant_columnNo attached table has that column. The message lists the attached tables and their columns.
404 no_sourceNo database is connected under that name. The message lists those that are.
429 too_many_runs, too_many_proposes, or “statements a minute”A limit from Rules and limits. A loop that never settles a conflict hits too_many_proposes.
Propose: 409 conflictRows changed in the database meanwhile. Proposing the same way again fails the same way: settle it ("by_columns": true first) or have a person approve it.
Approve: 409 staleThe run changed after the person saw it. Make a new approval link.
Approve: 409 reads_changedRows the run read changed after it began, or a table it read without a narrowing filter has more than 100,000 rows to check. Nothing was written. Reject it and start a new run, or approve with skip_read_check: true: the rows it changed are still checked.
Undo: 409 undo_not_readyThe gateway’s role was made before undo had its table. A database administrator runs the statements in error.sql once. Runs merged after that can be undone.
Approval link: 410 expired or 409 usedMake a new link with POST /v1/runs/:id/approvals.
The widget doesn’t load or its calls failThe browser must reach GATEWAY_PUBLIC_URL, and your site’s origin must be in GATEWAY_ALLOWED_ORIGINS.
The console says the page was opened at the wrong address (403 origin)The console only answers pages opened at GATEWAY_PUBLIC_URL. Set it, or GATEWAY_DOMAIN, to the address people use.
The gateway doesn’t start: “GATEWAY_STORE_URL needs GATEWAY_NODE_URL”Set how the other gateways reach this one.
The gateway doesn’t start: “GATEWAY_OIDC_ALLOW names who may approve”Single sign-on needs an allow list. Without it, anyone your provider signs in could approve.
Log warning “listening on 0.0.0.0:7400 over plain HTTP” in DockerExpected inside the container. Put TLS in front and set an https:// GATEWAY_PUBLIC_URL before exposing it.
Webhooks: pending keeps growingA receiver is down or slow. It must answer 2xx within 10 seconds.
MongoDB: GATEWAY_FOLLOW is on but changes arrive only on the timerThe role has no changeStream on the collections. engine.log says what is missing.

Threat model in brief

RuleHow
The agent never holds a credential to your databaseOnly the engine has the URL (a 600 file, never an argument). An agent gets a run token for the gateway.
The agent sees only its tenant’s rowsThe run’s agent is held to one tenant in the engine: every read and write of a table with the tenant column is filtered to it, whatever the agent’s SQL says.
Nothing lands without a check against live dataA merge is judged by the merge policy, then written in one transaction that locks and compares each row: one changed meanwhile stops it all.
Every change can be traced and undoneEach run’s world is tagged with its run and intent; the engine’s audit names the agent; an undo goes back through the database, checked the same way.
The data can stay in your networkThe gateway and engine run next to your database.
ThreatStopped byNot covered
Prompt injection makes the agent read or change another customer’s rowsThe engine’s tenant filter on every table with the tenant column, on reads and writes, and the world’s declared tenant at merge.A table with neither the tenant column nor a foreign key to one that has it is read whole by every run; writes to it are refused. Give it the column, or leave it out.
A run token leaksIt opens one run, for its time.Within its time, it can do what the run can.
The developer key leaksRevoke it. It never reaches the database directly.It can start runs and approve held ones: keep it in your backend only.
An approval link is forwardedIt decides one held change, once, as it was when made. With single sign-on, a link made for an approver (or every link, GATEWAY_OIDC_REQUIRE=on) also needs that person signed in.Without single sign-on, whoever holds it within its time can decide.
Someone signs in who shouldn’t approveGATEWAY_OIDC_ALLOW, checked at sign-in and on every request; the ID token’s signature, issuer, audience, expiry and nonce; PKCE.An account taken over at your provider.
Another site’s page acts with a console sign-inEach call needs the console’s header and an Origin equal to GATEWAY_PUBLIC_URL; SameSite=Lax; a strict content policy and no framing.
A new gateway is set up by someone else firstThe setup code is printed only to its log, works only while no key exists, once, for 7 days, and stops after 20 wrong tries.Someone who reads the log. Keep the console off the network until it is set up.
An agent splits a big job into small runsThe policy’s hourly limits (rows_per_hour = 5000 by default), counted per tenant across every run.
Retry loops, huge readsGATEWAY_RUN_STATEMENTS_PER_MINUTE, GATEWAY_RUN_MAX_ROWS, GATEWAY_RUN_PROPOSES_PER_MINUTE, GATEWAY_RUNS_PER_HOUR.
The gateway’s host is breachedThe store holds hashes and sealed engine tokens; the key file and the database URL are mode 600.A breach of the host is a breach of the gateway’s database role: make it least-privilege (role_sql).

Not there yet: SOC 2.

Configure and operate · Chronos