Gateway
Configure and operate
Every setting, tokens and what each can do, high availability, backups, upgrades and what to check when something is off.
Self-hostedChronos Gateway runs on your own servers today. Releases and install script. The engine under it: Chronos DB, self-hosted
This page is the reference for running the gateway: gateway 0.4.4 on Chronos DB (the engine) 0.6.2, image ghcr.io/abhishekxdg/chronos-gateway. Installing it is in Install and set up.
How settings are read
The gateway reads its settings from environment variables, once, when it starts. A setting that is wrong stops it with a sentence saying what it takes.
| Install | Where settings go | Apply them |
|---|---|---|
Installer, --service | /etc/chronos-gateway/gateway.env | sudo systemctl restart chronos-gateway |
| Docker | -e NAME=value on docker run | Remove the container and run it again |
| Docker Compose | .env beside compose.yaml | docker compose up -d |
| Helm | The chart’s values; env for any GATEWAY_* setting | helm upgrade |
- Times take
60,60s,15m,1hor1d. A bare number is seconds. - For limits,
0means no limit. - Switches take
onoroff(also1,true,yesand0,false,no). chronos-gateway --helplists the settings too.
Where it runs
| Variable | Default | Meaning |
|---|---|---|
| GATEWAY_LISTEN | 127.0.0.1:7400 (image: 0.0.0.0:7400) | Address and port it listens on. |
| GATEWAY_PUBLIC_URL | https:// + GATEWAY_DOMAIN, else http:// + GATEWAY_LISTEN (http://localhost:7400 when it listens on all addresses) | How clients and browsers reach it. Used in sql_url, mcp_url, approval links and the console, which only answers pages opened at this address. Use https:// anywhere but this machine. |
| GATEWAY_DOMAIN | unset | Its domain behind HTTPS (gateway.example.com, no https://, no path): sets GATEWAY_PUBLIC_URL to https://<domain> when that is unset. Compose’s https profile gets its certificate. |
| GATEWAY_DATA | gateway-data (service: /var/lib/chronos-gateway, image: /data) | Its data folder. |
| GATEWAY_ALLOWED_ORIGINS | none | Web origins allowed to call the approval endpoints from a browser, comma-separated (https://app.example.com). The sites that show the approval widget. |
| GATEWAY_API_KEY | unset | One more developer key to accept, besides those made with keys create. It can’t be revoked with keys revoke: remove it and restart. |
The engine and your database
| Variable | Default | Meaning |
|---|---|---|
| CHRONOS_BIN | the engine the installer put beside the gateway (<prefix>/lib/chronos-gateway/chronos), else chronos on the PATH | The engine binary the gateway starts. |
| CHRONOS_ENGINE_URL, CHRONOS_ADMIN_TOKEN | unset | Use an engine you run yourself instead of starting one. Both are needed. Then attach where that engine runs, and call POST /v1/setup. One database only, and not with several gateways. |
| GATEWAY_REFRESH_EVERY | 5m | How often the engine takes your database’s latest rows. Each refresh reads the attached tables in full. |
| GATEWAY_FOLLOW | off | on: the engine also takes the database’s changes about a second after they commit. See below. |
| CHRONOS_UNDO_WINDOW_DAYS | 30 | How many days after its merge a run can be undone. No upper limit; a longer window keeps more rows in the record table in your database. See Undo. |
| GATEWAY_SOURCE_FILE | unset | A file holding the database URL (a mounted secret). Copied into the data folder at each start, so a changed password is picked up at the next restart. While it is set, POST /v1/connect takes tables only and refuses a url. |
| GATEWAY_TABLES | unset | With GATEWAY_SOURCE_FILE: the tables to attach at the first start, comma-separated. |
| GATEWAY_TENANT_COLUMN | unset | With GATEWAY_TABLES: the tenant column. |
| GATEWAY_FIELDS | unset | With GATEWAY_SOURCE_FILE, beside MongoDB: the fields kept as columns, comma-separated. |
GATEWAY_FOLLOW, per database
| Beside | How it follows | Needs |
|---|---|---|
| Postgres | A logical replication slot | wal_level = logical and a role with REPLICATION. The connect check’s follow says what is missing. |
| MongoDB | A change stream on each attached collection | Engine 0.5.1 or later, and changeStream on those collections for the gateway’s role. Nothing to set up or clean up. |
| MySQL, MariaDB | Not yet | The gateway refreshes on its timer. |
The start-up settings (GATEWAY_SOURCE_FILE, GATEWAY_TABLES and the like) are for the first database only. One connected under a name is connected through the API or the console.
Rules and limits
| Variable | Default | Meaning |
|---|---|---|
| GATEWAY_POLICY | gateway | Name of the merge policy runs keep to. |
| GATEWAY_POLICY_RULES | max_rows = 500, max_deletes = 0, rows_per_hour = 5000, max_age = '15 minutes', check_reads = true, require_scope = true | The rules a change must keep to be written without a person. It replaces the whole list. Rules set in the console or with PUT /v1/policy are kept in the store and win over it. |
| GATEWAY_RUN_TTL | 1h | How long a run, and its token, lasts. A run can ask for less (ttl_seconds), not more. |
| GATEWAY_HOLD_TTL | 3d | How long a held change waits for a person before it expires. |
| GATEWAY_APPROVAL_TTL | 1d | How long an approval link works, at most. An undo link in a Slack message too. |
| GATEWAY_SLACK_UNDO | on | A Slack message of a merge carries a link that undoes it (once, for whoever holds it, rows changed since left alone). off: no link. |
| GATEWAY_RUN_STATEMENTS_PER_MINUTE | 120 | SQL statements, MCP tool calls and MongoDB tool calls per run per minute. Past it: 429. |
| GATEWAY_RUN_PROPOSES_PER_MINUTE | 6 | Proposes per run per minute. Each one locks and compares rows in your database. Past it: 429 too_many_proposes. |
| GATEWAY_RUNS_PER_HOUR | 60 | New runs per tenant per hour. Past it: 429 too_many_runs, and a run.refused event. |
| GATEWAY_RUN_MAX_ROWS | 1000 | Rows one SQL answer returns to an agent. Longer answers are cut and marked truncated. |
The default rules, in plain words: a change is written without a person only if it deletes no rows, changes at most 500 rows, keeps its tenant under 5,000 changed rows an hour (counted across all its runs), was made within the last 15 minutes, every row it read still matches the database, and it declares its tenant (every gateway run does). Anything else is held.
The rules are applied when you connect. To change them with the variable later, set it, restart, then apply it:
curl -s -X POST "$GATEWAY/v1/setup" -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d '{}'Effect delivery
From 0.4.0. With these set, the gateway delivers what merges queue in the engine’s outbox (NOTIFY ON MERGE). How it works is in Effects.
| Variable | Default | Meaning |
|---|---|---|
| GATEWAY_EFFECTS | none: effects are not delivered | channel=URL pairs, comma separated, * for any other channel: billing=https://billing.internal/hook,*=https://all.internal/hook. Channel names are letters, digits, _, - and .; receivers are http(s) URLs. |
| GATEWAY_EFFECTS_SECRET | none | Signs each delivery (Chronos-Signature). Required with GATEWAY_EFFECTS: the gateway won’t start without it. |
| GATEWAY_EFFECTS_MAX_ATTEMPTS | 8 | Tries before an effect is given up as dead. At least 1. |
Single sign-on
Unset by default: approval links work by themselves. Set these so people sign in with your OpenID Connect provider to approve and to use the console.
| Variable | Default | Meaning |
|---|---|---|
| GATEWAY_OIDC_ISSUER | unset | Your provider’s issuer URL (https://acme.okta.com, https://login.microsoftonline.com/<tenant>/v2.0, https://accounts.google.com). https:// only, or plain http:// to this machine. Register <GATEWAY_PUBLIC_URL>/sso/callback as the redirect URI. |
| GATEWAY_OIDC_CLIENT_ID, GATEWAY_OIDC_CLIENT_SECRET | unset | The gateway’s client at your provider. Both are needed. The secret is sent with HTTP Basic (client_secret_basic). |
| GATEWAY_OIDC_ALLOW | unset, required | Who may approve: emails (ada@acme.com), domains (@acme.com) and groups (group:db-approvers), comma-separated. Without it the gateway doesn’t start. |
| GATEWAY_OIDC_ADMINS | unset | Who may sign in to the console and change settings, in the same form. They may also approve. |
| GATEWAY_OIDC_GROUPS_CLAIM | groups | The ID token’s claim that lists a person’s groups (roles with Entra ID app roles). |
| GATEWAY_OIDC_REQUIRE | off | on: every approval link needs a person on the allow list, signed in. Off: only links made for an approver. |
| GATEWAY_OIDC_SESSION_TTL | 12h | How long a sign-in lasts. |
AI reviewer
A model decides each held change before a person does. Unset: people decide them all.
| Variable | Default | Meaning |
|---|---|---|
| GATEWAY_REVIEWER_URL | unset | An OpenAI-compatible API’s base: https://api.openai.com/v1, http://localhost:11434/v1 (Ollama), http://vllm:8000/v1. Plain http only to this machine, or without a key. |
| GATEWAY_REVIEWER_MODEL | unset, required | The model to ask. Any model works, hosted or open-weight. |
| GATEWAY_REVIEWER_KEY | unset | Its API key, sent as a bearer token. |
| GATEWAY_REVIEWER_MODEL_2 | unset | A second model on the same API. Both must approve, at the confidence below, for a change to merge. |
| GATEWAY_REVIEWER_MIN_CONFIDENCE | 0.8 | From 0 to 1. An approval less sure than this goes to a person. |
| GATEWAY_REVIEWER_INSTRUCTIONS | unset | Your rules for its decisions, in words. |
| GATEWAY_REVIEWER_INSTRUCTIONS_FILE | unset | A file to read those instructions from. It wins over the variable. |
| GATEWAY_REVIEWER_MAX_ROWS | 100 | The most changed rows it is shown. |
| GATEWAY_REVIEWER_COLUMNS | unset | Columns whose text the model reads as it is: leads.name,leads.stage, leads.*, or *. Other text it sees only as a placeholder. |
| GATEWAY_REVIEWER_AGENT | gateway_reviewer | The engine agent it decides as: letters, digits and _. |
The reviewer works for the first database only. What it is shown and what it can’t approve are in Rules, labels and masks.
Link to Chronos Cloud
Optional, and unset by default: then nothing below runs. With GATEWAY_CLOUD set, the gateway opens one outbound TLS connection to Chronos Cloud so its dashboard can show reviews and send back what people decide there. The database URL and the engine’s admin token never cross that link. There is no inbound port.
| Variable | Default | Meaning |
|---|---|---|
| GATEWAY_CLOUD | unset | The cloud’s https:// address. |
| GATEWAY_CLOUD_ENROLL | unset | The one-time enrollment token (cce_...) from the dashboard, for the first start only. |
| GATEWAY_CLOUD_ALLOW | read,merge,discard,undo | What the cloud may ask for. Also agents, policy, sql (read-only). Without merge, merging stays local. |
| GATEWAY_CLOUD_SHARE | diffs | diffs: changed rows, less the columns your labels hide; masked columns read masked. counts: no row leaves your network. Either way a cloud diff never returns SQL and a refusal loses its detail (from 0.4.3), and a cloud merge of a run’s world is an approval: of a run held or in conflict, at the version the person saw, into main. |
| HTTPS_PROXY, NO_PROXY | unset | A proxy for that link (HTTP CONNECT), and hosts that skip it. |
Several gateways
| Variable | Default | Meaning |
|---|---|---|
| GATEWAY_STORE_URL | unset | A Postgres database for the gateway’s own tables, shared by every gateway. Unset: the store is SQLite in the data folder, and there is one gateway. |
| GATEWAY_NODE_URL | unset | With GATEWAY_STORE_URL, required: how the other gateways reach this one (http://10.0.0.5:7400). |
| GATEWAY_SECRET_FILE | <data>/secret.key | With GATEWAY_STORE_URL: the file of the key that seals the store’s secrets. The same for every gateway sharing the store. |
See High availability for what these give and need.
Logs and tracing settings
| Variable | Default | Meaning |
|---|---|---|
| RUST_LOG | chronos_gateway=info | Log detail, for example chronos_gateway=debug. |
| NO_COLOR | unset | Set: no colour codes in the log. Under systemd and Docker (no terminal) there are none anyway. |
| OTEL_EXPORTER_OTLP_ENDPOINT | unset | An OpenTelemetry collector, or a vendor’s OTLP/HTTP endpoint. Unset, nothing is sent. |
| OTEL_EXPORTER_OTLP_HEADERS | unset | Headers for it, such as a vendor’s key. |
| OTEL_SERVICE_NAME | chronos-gateway | The service’s name in traces. OTEL_RESOURCE_ATTRIBUTES and the other standard OTEL_* variables work as usual. |
With tracing on, the gateway sends a span for each request, named by its route (POST /v1/runs/:id/approve) with its status, a span for each call to the engine inside it, and one for each webhook delivery. A request with a W3C traceparent header joins the caller’s trace.
The command line
| Command | Does |
|---|---|
| chronos-gateway | Runs the gateway, the same as serve. Settings come from the environment. |
| chronos-gateway serve | Runs the gateway. Ctrl-C or SIGTERM stops it: the engines are stopped cleanly first. |
| chronos-gateway keys create <name> | Makes a developer key and prints it (cgk_...), once. Its id goes to standard error. |
| chronos-gateway keys list | One line per key: id, name, active or revoked. |
| chronos-gateway keys revoke <id> | Revokes a key. A console sign-in made with it ends too. |
| chronos-gateway setup-code | A new code to set the gateway up in the browser, while it has no key yet. It works once, for a week. |
| chronos-gateway --help | The usage and the settings. Also -h and help. |
| chronos-gateway --version | The version. Also -V. |
keys and setup-code open the store, so run them with the same GATEWAY_DATA (and GATEWAY_STORE_URL, if set) and as the same user as the gateway:
| Install | Make a key |
|---|---|
Installer, --service | sudo -u chronos-gateway env GATEWAY_DATA=/var/lib/chronos-gateway /usr/local/bin/chronos-gateway keys create backend |
| Docker | docker exec chronos-gateway chronos-gateway keys create backend |
| Docker Compose | docker compose exec gateway chronos-gateway keys create backend |
| Helm | kubectl -n chronos exec statefulset/chronos-gateway -- chronos-gateway keys create backend |
An unknown command prints the usage and exits with 2. A command that fails prints chronos-gateway: ... and exits with 1.
Files in the data folder
The data folder (GATEWAY_DATA) is readable by the gateway’s user only; the service makes it mode 700.
| File | Holds |
|---|---|
| gateway.db | The SQLite store: token hashes, runs, approvals, events, and webhook signing secrets (sealed from 0.4.4, with secret.key; those made before are kept as they are). Mode 600. Not there with GATEWAY_STORE_URL. |
| secret.key | The key that seals the engine tokens of run agents and webhook secrets inside the store. Mode 600. |
| source.url | The database URL, with its password. Mode 600. |
| engine.json | The engine’s local port and admin token. Mode 600. |
| db/ | The engine’s folder: the copy of the attached tables and the runs’ worlds. |
| engine.log | The engine’s output. |
| sources/<name>/ | The same files for a database connected under a name. |
| cloud.key, cloud.crt, cloud-ca.crt, cloud.json, cloud.hmac | Only when linked to Chronos Cloud: the connector’s key (600), its certificate, the cloud’s CA, the link’s address, and the key that hashes row ids under counts (600). |
Tokens
Each token is 32 random bytes with a prefix. The gateway stores only their SHA-256 hashes.
| Token | Who holds it | What it opens | How long |
|---|---|---|---|
| cgk_ developer key | your backend | Everything except a run’s own SQL and MCP, and the widget’s calls. It can start runs and approve held changes. | Until revoked (keys revoke <id>, the console, or DELETE /v1/keys/:id). |
| cgr_ run token | one agent | That one run: its SQL, MCP, MongoDB tools, changes, status and propose. | Until the run ends or its time runs out (GATEWAY_RUN_TTL). |
| cga_ approval token | one person | One held change, at the version it was made for, once. | GATEWAY_APPROVAL_TTL. |
| cgu_ undo link | whoever is in the Slack channel | Undo one merged change, once, as far as its rows are unchanged. | GATEWAY_APPROVAL_TTL. |
| setup code | whoever reads the log | Makes the first key, in the console. Works only while no key exists. | Once, for 7 days; stops after 20 wrong tries. |
| sign-in cookie | a person’s browser | The console (a key, or single sign-on on GATEWAY_OIDC_ADMINS), or approving (GATEWAY_OIDC_ALLOW). | GATEWAY_OIDC_SESSION_TTL (12 hours). A key’s sign-in ends when the key is revoked. |
- Keep the developer key in your backend only. It never reaches your database directly, but it can start runs and approve held ones.
- Make one key per backend, so each can be revoked alone.
- The agent never holds a credential to your database. Behind a run token is an engine agent held to one tenant, whose own token works for the run’s time only.
- The sign-in cookie is
HttpOnly,SameSite=Lax, andSecureover https. It is kept as its hash. A person is checked against the allow list on every request, so removing someone fromGATEWAY_OIDC_ALLOWand restarting ends their access.
What reaches your database
| When | What happens in your database |
|---|---|
POST /v1/connect/check | The gateway logs in with the URL you sent and reads the catalog. No writes. The URL isn’t kept. |
POST /v1/connect | The engine logs in as the gateway’s role and reads the attached tables to copy them. The copy holds one read open while it runs. |
Every GATEWAY_REFRESH_EVERY | The engine reads the attached tables again. |
Every second, with GATEWAY_FOLLOW | The engine reads the changes since the last second from its replication slot (MongoDB: a change stream on each attached collection), then the rows they name. |
| An agent’s SQL, MCP or MongoDB tool call | Nothing. It runs in the run’s world. |
| A merge (rules kept, or a person approved) | One transaction: each row the merge writes is locked and compared with the database, then written, and the merge is recorded in chronos.merges. If any row changed meanwhile, nothing is written. |
| An undo | One transaction of the same kind: each row locked and compared, then put back as it was. |
| Held, rejected, expired or discarded | Nothing. |
- The engine’s own tables are
chronos.mergesandchronos.merge_rowsbeside Postgres,chronos_mergesandchronos_merge_rowsbeside MySQL and MariaDB, and collections of those names beside MongoDB.merge_rowskeeps what each merge wrote, for undo. - Your app’s reads and writes never pass through the gateway.
- With
act_as, a run’s writes are made afterSET LOCAL ROLE, as your end user’s role (Postgres only). - Between refreshes an agent may read a row that is out of date. That is safe: the merge compares every row it writes, and
check_reads = trueholds a change if a row it read has changed.
On the network
- The engine listens on
127.0.0.1only. - The gateway listens on
127.0.0.1:7400unless you change it, over plain HTTP. Tokens travel in headers, so put TLS in front (nginx, Caddy, your load balancer) before anything on another machine calls it, and setGATEWAY_PUBLIC_URLto thehttps://address. - The gateway logs a warning when it listens beyond loopback with an
http://public URL. In the image that is expected: it is fine while the port is published to127.0.0.1only. - Outbound, the gateway reaches your database, and only what you set up: webhook targets, your sign-on provider, an AI reviewer’s API, an OpenTelemetry collector, Chronos Cloud.
- The Helm chart’s
networkPolicy.enabled(off by default) lets the pod take calls on 7400 from its own namespace and reach only your database, DNS, and port 443 on the addresses you list.
High availability
One gateway on one data folder is the default, kept available by restarting it fast on the same data: the service unit restarts it on failure, and in containers one replica runs with its data on a volume that can move to another node. Since 0.3.0 several gateways can also run behind one address. One leads at a time; when it stops or is lost, another takes over.
What it gives, and what it doesn’t
- The API stays up through the loss of a gateway or its node. A request may go to any gateway.
- After a leader is lost, another leads within about 15 seconds (measured: 13 s after
kill -9, 4 s after a clean stop). Requests in those seconds answer 503no_leaderwithRetry-After: 5. - Merges pause for those seconds. A merge that was in flight is settled by the engine from the database’s own record of merges, as after any crash: it either happened whole or not at all.
- A run that was open when the leader died hard loses its work. See Durability. A clean handover (a rolling update,
SIGTERM) keeps open runs. - It does not make the database agents work on highly available: that is yours.
What it needs
- A store in Postgres, shared by every gateway:
GATEWAY_STORE_URL=postgres://.... A database for the gateway’s own tables (keys, runs, approvals, events, the lease). It can live on the server agents work on, in a database of its own. - One data folder, mounted by every gateway at
GATEWAY_DATA: a ReadWriteMany volume (EFS, Filestore, Azure Files, CephFS, NFS). It holds the engines’ folders, the database URLs and the key that seals the store’s secrets. Only the leader’s engines write it. - A node URL:
GATEWAY_NODE_URLon each gateway: how the others reach it (http://10.0.0.5:7400). The leader’s goes into its lease, and followers send requests there. Allow that traffic between the gateways. - The engine run by the gateway itself, not
CHRONOS_ENGINE_URL.
If the gateways don’t share a data folder’s secret.key, give each the same key file with GATEWAY_SECRET_FILE. A gateway with GATEWAY_STORE_URL and no GATEWAY_NODE_URL doesn’t start.
How it works
| Part | How |
|---|---|
| The lease | A row in the store (leases), taken in one statement, renewed every 5 seconds, good for 15. Time is the store’s clock, not each machine’s. |
| The leader | Runs the engines and the work that must happen once: the sweeper, webhook deliveries, the model reviewer, Chronos Cloud’s link. |
| A follower | Answers /health itself and sends every other request to the leader, unchanged, and returns the leader’s answer. The engines stay on the leader’s loopback. |
| Taking over | A leader that stops gives the lease up. One that dies stops renewing; after 15 seconds a follower takes the lease and starts the engines on the shared folder. |
| Two engines never share a folder | A leader that can’t renew for 8 seconds (the store unreachable, or the lease taken) stops its engines, kills any still there 4 seconds later, and exits: all before its 15-second lease runs out. Whatever runs it starts it again, as a follower. Measured with the store frozen: engine gone and gateway exited after 6.5 s. On Linux an engine is also told to stop when its gateway dies. |
| Timing | Value |
|---|---|
| Lease renewed | every 5 seconds |
| Lease good for | 15 seconds |
| A leader gives up when it can’t renew for | 8 seconds |
| Engines still running are killed after | 4 more seconds |
Retry-After on 503 no_leader | 5 seconds |
| A follower’s request to the leader | connects within 3 seconds, waits up to 150 seconds |
| A statement the store doesn’t answer | fails its request after 30 seconds |
GET /v1/status says ha: {shared, node, role, leader: {node, url, expires}}. Point your load balancer’s health check at GET /health, which every gateway answers itself.
Kubernetes
ha: enabled: true replicas: 2 storeExistingSecret: chronos-store # key storeUrl existingClaim: chronos-gateway-shared # ReadWriteManyThe chart sets GATEWAY_NODE_URL from each pod’s address. With a NetworkPolicy, allow the gateways’ pods to reach each other on 7400.
Limits
- One connection to the store’s Postgres per gateway, one statement at a time.
- A follower holds a request’s body in memory to send it on (64 MB at most).
- The fence between an old leader and a new one is time (the lease) and, on Linux, the engine stopping with its gateway. A leader paused whole for longer than 15 seconds and then resumed (a frozen VM) runs for a fraction of a second more before its watch notices.
- Not built: taking the engine’s folder over from object storage (so no shared volume is needed), a pool of store connections, moving an existing SQLite store into Postgres.
Durability
| What | Through a clean stop | Through a hard crash |
|---|---|---|
| A merge already written | Kept: it is in your database. | Kept. |
| A merge in flight | The gateway stops the engine cleanly before it exits. | Settled when the engine starts, from the database’s record of merges (chronos.merges). Nothing is written twice and nothing is lost. |
| Runs, holds, approval links, events, keys | Kept in the store. | Kept in the store. |
| An open run’s unmerged work | Kept. | Not kept. |
The engine keeps a world’s unmerged changes through a clean stop, not through a crash. After a hard crash (a kill -9, a lost node, a power cut) the next call on a run whose world was lost answers 409 world_lost. The run becomes failed, run.failed is sent with code: "world_lost", and nothing of it has reached the database. Start a new run. This holds for one gateway as much as for several.
So build the agent’s side to be able to start again: a run is one task, and a task that fails is run again from the top. GATEWAY_RUN_TTL (1 hour) already bounds how long any run’s work lives unmerged.
Backups
Back up the data folder along with your database. It holds the store, the keys and the engine’s folder. For a consistent copy, stop the gateway first:
sudo systemctl stop chronos-gatewaysudo tar -C /var/lib -czf chronos-gateway-backup.tar.gz chronos-gatewaysudo systemctl start chronos-gateway- With
GATEWAY_STORE_URL, the store is in Postgres: back that database up as you back up any other. - The backup holds
source.urlandsecret.key. Keep it as carefully as the database password. - Your data is in your database, not here. A lost data folder loses the gateway’s keys, runs waiting for approval, events and settings, not rows your app owns.
- Agents can’t start runs or merge while the gateway is down. Your database and your app aren’t affected: the gateway is never in their path.
Upgrades and versions
Back up the data folder first. Upgrades keep settings, data and keys.
| Install | Upgrade |
|---|---|
Installer, --service | Run the install command again. It replaces both binaries and restarts the service. |
| Installer, binaries only | Run the install command again, then restart your chronos-gateway. |
| Docker | docker pull ghcr.io/abhishekxdg/chronos-gateway:<version>, then docker rm -f chronos-gateway and run the same docker run command with the new tag. The volume carries over. |
| Docker Compose | Set the new GATEWAY_VERSION in .env, then docker compose pull && docker compose up -d. |
| Helm | helm upgrade chronos-gateway <the new release’s chart URL> -n chronos --reuse-values. The volume carries over. |
Check the result with chronos-gateway --version. The bundled engine answers --version too:
| Install | Engine version |
|---|---|
Installer, --service | /usr/local/lib/chronos-gateway/chronos --version |
| Installer, binaries only | ~/.local/lib/chronos-gateway/chronos --version |
| Docker | docker exec chronos-gateway chronos --version |
| Helm | kubectl -n chronos exec statefulset/chronos-gateway -- chronos --version |
The engine a feature needs
Each gateway release bundles the engine it was tested with: gateway 0.4.4, 0.4.3, 0.4.2 and 0.4.1 with engine 0.6.2, gateway 0.4.0 with engine 0.6.0, gateway 0.3.4 with engine 0.5.5, gateway 0.3.3 with engine 0.5.3, gateway 0.3.2 with engine 0.5.2, gateway 0.3.1 with engine 0.5.1, gateway 0.3.0 with engine 0.5.0. The installer and the image install both together, so this matters only if you set CHRONOS_BIN or CHRONOS_ENGINE_URL to an engine of your own.
| Feature | Engine |
|---|---|
| Masks, tables with row-level security on | 0.4.4 or later |
| Runs change child tables (reached through a foreign key) | 0.4.5 |
| A run is held only by its own tenant’s rows | 0.4.6 |
act_as: runs write as your end user | 0.4.9 |
The reviewer’s policy rules (trusted_columns, watch_totals, canary, ...) | 0.4.10 |
| A run’s reads checked against MySQL and MariaDB | 0.4.11 or later. With an older engine the default rules leave check_reads off; after upgrading it, POST /v1/setup turns the check on. |
Rows added to AUTO_INCREMENT tables | 0.4.12 or later |
| Undo, MongoDB | 0.5.0 or later (MongoDB: an engine built with it) |
GATEWAY_FOLLOW beside MongoDB | 0.5.1 or later |
Evidence attached to the run’s world (in_engine: true), from gateway 0.4.0 | engine 0.6.0 (ATTACH EVIDENCE). With an older engine the evidence is kept with the run only |
- A role made from the statements of 0.3.0 or earlier has no table for undo. Merges work;
GET /v1/statussaysattached.undo.ready: false, and an undo answers 409undo_not_readywith the statements to run once. - A MongoDB role made from older statements has no
changeStream.GATEWAY_FOLLOWthen keeps refreshing on its timer until it is granted. - The 0.2.11 image can’t start its engine. Use 0.3.0 or later.
- Each binary and the image are signed with cosign (since 0.2.6).
CHRONOS_VERIFY=1makes the installer check both.
Health and logs
| Check | How |
|---|---|
| Is it up | GET /health answers {"ok": true} without a token. For load balancers and probes. |
| Is the engine healthy | GET /v1/status with a key: engine.healthy. |
| Are webhooks getting through | GET /v1/status: webhooks.pending and webhooks.failed. |
| Who leads | GET /v1/status: ha. |
| The gateway’s log | Service: journalctl -u chronos-gateway. Docker: docker logs chronos-gateway. Compose: docker compose logs gateway. |
| The engine’s log | engine.log in the data folder (/var/lib/chronos-gateway/engine.log; in Docker, docker cp chronos-gateway:/data/engine.log .). |
| An audit trail | Every run, hold, approval, merge and rejection is an event. Point a webhook at your log pipeline, or read GET /v1/events. |
- The gateway starts the engine again if it stops, and stops it cleanly on shutdown.
- The service unit runs as
chronos-gatewaywith a hardened sandbox (ProtectSystem=strict,NoNewPrivileges, write access to the data folder only), and restarts on failure. - The image runs as user 65532 on a distroless base, with no shell. Compose and the Helm chart add a read-only root filesystem and no Linux capabilities.
- On big tables, refresh less often (
GATEWAY_REFRESH_EVERY), or turn onGATEWAY_FOLLOW.
Troubleshooting
| What you see | Why, and what to do |
|---|---|
401 with “send a developer key” | The Authorization: Bearer cgk_... header is missing, or the token is a run or approval token. |
/v1/connect/check: “not a Postgres URL: invalid value for option sslmode” or “unknown option” | The check takes sslmode disable, prefer or require, and no other URL options. Drop verify-full, sslrootcert and the like for the check only. /v1/connect takes them. |
/v1/connect/check: a certificate error | Only sslmode=verify-ca or verify-full checks the certificate, against the public web CAs. With a private CA or a self-signed certificate, use sslmode=require for the check. |
/v1/connect: “about N rows in all ... Send yes: true to go on” (400 confirm) | Over 1,000,000 rows, attaching asks first: the copy holds one read open on the database while it runs (2.5 million rows took about 30 s on a test machine). Send the same request with "yes": true. |
/v1/connect: “the server asks for the password as MD5” (or “in the clear”) | Set password_encryption = 'scram-sha-256' and set the role’s password again, or connect with sslmode=verify-full. |
/v1/connect: “no CA certificates found” | verify-full found no system CA bundle. Add sslrootcert=<file.pem> with your CA, readable by the gateway’s user (in Docker, mount it). |
/v1/connect in Docker can’t reach a database on the same machine | Inside the container, localhost is the container. Use host.docker.internal (on Linux add --add-host=host.docker.internal:host-gateway). |
/v1/connect: 409 attached | This gateway has a database already. Connect another under a name, or use a new data folder. |
| “the engine didn’t start within 30 s” | Read engine.log in the data folder. |
503 no_engine | No database is connected yet: POST /v1/connect. |
503 no_leader | With several gateways: none leads right now. Try again after Retry-After. If it lasts, check that the gateways reach the store and each other (GATEWAY_NODE_URL). |
409 world_lost on a run | The engine crashed and the run’s unmerged work went with it. Start a new run. |
POST /v1/runs: 400 tenant_column | No attached table has that column. The message lists the attached tables and their columns. |
404 no_source | No database is connected under that name. The message lists those that are. |
429 too_many_runs, too_many_proposes, or “statements a minute” | A limit from Rules and limits. A loop that never settles a conflict hits too_many_proposes. |
Propose: 409 conflict | Rows changed in the database meanwhile. Proposing the same way again fails the same way: settle it ("by_columns": true first) or have a person approve it. |
Approve: 409 stale | The run changed after the person saw it. Make a new approval link. |
Approve: 409 reads_changed | Rows the run read changed after it began, or a table it read without a narrowing filter has more than 100,000 rows to check. Nothing was written. Reject it and start a new run, or approve with skip_read_check: true: the rows it changed are still checked. |
Undo: 409 undo_not_ready | The gateway’s role was made before undo had its table. A database administrator runs the statements in error.sql once. Runs merged after that can be undone. |
Approval link: 410 expired or 409 used | Make a new link with POST /v1/runs/:id/approvals. |
| The widget doesn’t load or its calls fail | The browser must reach GATEWAY_PUBLIC_URL, and your site’s origin must be in GATEWAY_ALLOWED_ORIGINS. |
The console says the page was opened at the wrong address (403 origin) | The console only answers pages opened at GATEWAY_PUBLIC_URL. Set it, or GATEWAY_DOMAIN, to the address people use. |
| The gateway doesn’t start: “GATEWAY_STORE_URL needs GATEWAY_NODE_URL” | Set how the other gateways reach this one. |
| The gateway doesn’t start: “GATEWAY_OIDC_ALLOW names who may approve” | Single sign-on needs an allow list. Without it, anyone your provider signs in could approve. |
| Log warning “listening on 0.0.0.0:7400 over plain HTTP” in Docker | Expected inside the container. Put TLS in front and set an https:// GATEWAY_PUBLIC_URL before exposing it. |
Webhooks: pending keeps growing | A receiver is down or slow. It must answer 2xx within 10 seconds. |
MongoDB: GATEWAY_FOLLOW is on but changes arrive only on the timer | The role has no changeStream on the collections. engine.log says what is missing. |
Threat model in brief
| Rule | How |
|---|---|
| The agent never holds a credential to your database | Only the engine has the URL (a 600 file, never an argument). An agent gets a run token for the gateway. |
| The agent sees only its tenant’s rows | The run’s agent is held to one tenant in the engine: every read and write of a table with the tenant column is filtered to it, whatever the agent’s SQL says. |
| Nothing lands without a check against live data | A merge is judged by the merge policy, then written in one transaction that locks and compares each row: one changed meanwhile stops it all. |
| Every change can be traced and undone | Each run’s world is tagged with its run and intent; the engine’s audit names the agent; an undo goes back through the database, checked the same way. |
| The data can stay in your network | The gateway and engine run next to your database. |
| Threat | Stopped by | Not covered |
|---|---|---|
| Prompt injection makes the agent read or change another customer’s rows | The engine’s tenant filter on every table with the tenant column, on reads and writes, and the world’s declared tenant at merge. | A table with neither the tenant column nor a foreign key to one that has it is read whole by every run; writes to it are refused. Give it the column, or leave it out. |
| A run token leaks | It opens one run, for its time. | Within its time, it can do what the run can. |
| The developer key leaks | Revoke it. It never reaches the database directly. | It can start runs and approve held ones: keep it in your backend only. |
| An approval link is forwarded | It decides one held change, once, as it was when made. With single sign-on, a link made for an approver (or every link, GATEWAY_OIDC_REQUIRE=on) also needs that person signed in. | Without single sign-on, whoever holds it within its time can decide. |
| Someone signs in who shouldn’t approve | GATEWAY_OIDC_ALLOW, checked at sign-in and on every request; the ID token’s signature, issuer, audience, expiry and nonce; PKCE. | An account taken over at your provider. |
| Another site’s page acts with a console sign-in | Each call needs the console’s header and an Origin equal to GATEWAY_PUBLIC_URL; SameSite=Lax; a strict content policy and no framing. | |
| A new gateway is set up by someone else first | The setup code is printed only to its log, works only while no key exists, once, for 7 days, and stops after 20 wrong tries. | Someone who reads the log. Keep the console off the network until it is set up. |
| An agent splits a big job into small runs | The policy’s hourly limits (rows_per_hour = 5000 by default), counted per tenant across every run. | |
| Retry loops, huge reads | GATEWAY_RUN_STATEMENTS_PER_MINUTE, GATEWAY_RUN_MAX_ROWS, GATEWAY_RUN_PROPOSES_PER_MINUTE, GATEWAY_RUNS_PER_HOUR. | |
| The gateway’s host is breached | The store holds hashes and sealed engine tokens; the key file and the database URL are mode 600. | A breach of the host is a breach of the gateway’s database role: make it least-privilege (role_sql). |
Not there yet: SOC 2.