Chronos DB 0.4.2
Joins, merges and memory, from running BranchBench (Columbia's benchmark of agent workflows on branching databases): joins on several columns and joins too big for memory, merges where both sides changed a table's columns, a merge that counts its conflicts, `DIFF ... SUMMARY` and `UNDO DIFF`, and a memory limit for every query
Shipped with Chronos Gateway 0.1.1, Chronos Gateway 0.1.0.
Joins, merges and memory, from running BranchBench (Columbia's benchmark of agent workflows on branching databases): joins on several columns and joins too big for memory, merges where both sides changed a table's columns, a merge that counts its conflicts, DIFF ... SUMMARY and UNDO DIFF, and a memory limit for every query.
Changed from 0.4.1: every statement has a memory limit. A statement past CHRONOS_QUERY_MEM (default a quarter of the machine's memory; 0 for none, as before) fails alone with 53200, and statements together are held to twice that, where a runaway join used to take the server down. See the entry below.
- Joins on several columns too big for memory hash on all of them too. A hash join whose other side outgrows
CHRONOS_WORK_MEMgoes to disk, split into files by the join key; that split, and each file's hash table, used the first equality alone. On TPC-C's(w_id, d_id, ...)keys that put a warehouse's rows in one file and paired each with all the others there: CH Q3 and Q18 took 17 minutes at five warehouses (16 MB of work memory: at one), where they now take 0.6 s. Found by BranchBench. - Joins take the equalities their ON clauses imply.
customer c JOIN new_order n ON n.w = c.w AND n.d = c.d JOIN orders o ON o.w = n.w AND o.d = n.d AND o.id = n.o AND o.c = c.id(TPC-C's CH Q3) joined customer to new_order first, as written: every new order of a district met every customer of it, and the rows held grew past 12 GB at one warehouse (at five, the server was killed for memory). Equal columns now carry across (o.w = n.wandn.w = c.wgiveo.w = c.w, between columns of one type and collation), and the next table joined is the one linked by its whole key, then by the most columns: orders comes second. Q3 at one warehouse: 0.35 s and 850 MB. Found by BranchBench. - Joins on several columns hash on all of them. A join on
(w_id, d_id, o_id)built its hash table on the first equality alone and checked the others row by row, so where that column has few values (one warehouse) every row probed the whole warehouse: TPC-C'sorder_line JOIN ordersat one warehouse took over two minutes,orders LEFT JOIN new_order4 s,customer JOIN orders27 s (Postgres: 74 ms, 3 ms, 11 ms). An in-memory hash join now keys on everya.x = b.yof the same kind; the same three take 56 ms, 14 ms and 15 ms. A join bigger thanCHRONOS_WORK_MEM(on disk) still hashes on one column. Found by BranchBench, whose TPC-C consistency checks are such joins. - MCP tells agents to settle conflicts by column first. The instructions used to say "decide which side wins and call merge again with resolve set to ours or theirs", and agents did, overwriting main's changes to columns they never touched. They now say that merging the same way again fails the same way, and give the safest order:
merge_preview, thenby_columns=true(withresolveto pick a side only for the columns both changed), thenrowsfor single rows, andresolvealone, which takes whole rows, last. A conflict frommergesays the same, also when the call already hadby_columns,rowsor the like (before, only a plainmergegot the hint), and for a conflict with an attached Postgres. Theresolveandby_columnsdescriptions and the TypeScript and Python clients'mergedocs say what each does to the other side's columns. - The log ships to cloud storage as it is written. With
--remote, a database's log used to stay on local disk until the next checkpoint (every 64 MB of log), so losing the machine lost everything since. It now goes to the bucket in chunks (wal-<segment>-<offset>objects) withinwal_shipping_delay(1 s) of being written, and a--takeoverfrom an empty folder on another machine replays it.ALTER SYSTEM SET wal_shipping = syncmakes a commit tomainor a merge return only once its log is in the bucket, so a takeover loses no acknowledged write;offships only at checkpoints, as before. A takeover fences the old writer at the end of the shipped log: its next upload fails and it ships nothing more. Checkpoints delete the shipped log the bucket no longer needs. Also new:gs://bucket/prefixremotes (Google Cloud Storage). See operations and the design. USE WORLD 'name@when'switches a session to a world as it was. The docs saiduse world 'main@2026-09-20 10:00'reads the past for the rest of the session, as connecting to the databasemain@...does, butUSE WORLD(andSWITCH WORLD) only took a name or an ID and failed withbranch "main@..." does not exist. It now takes'name@when'too: read only (writes fail with 25006), 22023 for a time outside the history window, 3D000 for a world that doesn't exist. Found by BranchBench, whose commit refs read a branch at a commit.DIFFsees a column's new size.ALTER TABLE t ALTER COLUMN v TYPE varchar(10)on avarchar(5)(ornumeric(10,2)tonumeric(12,2),integertosmallint,texttovarchar(20)) showed inDIFFas analter tablewith no column named incolumns, andDIFF ... AS SQLleft the change out, so running it elsewhere kept the old size. Both compare the whole type now, size included:v: varchar(5) -> varchar(10), andalter table t alter column v type varchar(10);. Found by BranchBench, whose schema diffs readcolumns.- Attach: rows Postgres triggers write to other tables come into main with the merge. A trigger on
leadsthat keepsaccounts.lead_count, writes an audit row, or deletes a lead's notes (and a foreign key'sON DELETE CASCADE) left those rows as main had them until the next refresh, and a world that changed one of them meanwhile got a conflict. The write-back now counts what its transaction wrote per attached table (deferred triggers included), and the tables written besides the merge's own rows are refreshed alone right after it; after a crash, recovery does that refresh. REINDEX DATABASErebuilds the database's indexes. It reported success and rebuilt nothing (it was read asREINDEX SYSTEM). It now rebuilds every index of the tables and materialized views you own, moves them withREINDEX (TABLESPACE t) DATABASE, and, as in Postgres, can't run inside a transaction block.REINDEX SYSTEMstill rebuilds only the catalogs'.- Temporary tables follow
temp_tablespaces. A temporary table or index made withoutTABLESPACEwent wheredefault_tablespacesaid, so aCREATE INDEXon one failed with 42501 when that named a tablespace you can't create in. It now goes to the first tablespace oftemp_tablespacesyou may create in, else the database's default, as in Postgres.CREATE TEMP TABLE ... TABLESPACE tworks (it saidtablespace "t" does not exist), andDROP TABLESPACErefuses one that holds the session's temporary tables or indexes. BY COLUMNS USING OURS(orTHEIRS) settles a row both sides changed column by column. The side named wins only the columns both changed; a column only the other side changed keeps its value. It used to take the named side's whole row, putting back the fork's value in columns only the other side changed (anemailmain changed went back to what it was). The same goes forUSING OURS|THEIRSinto a table set tomerge_by_columns.USING OURSwithoutBY COLUMNSstill takes whole rows. Under a merge policy, only a column ours actually overwrites counts as an overwrite.MERGE ... DRY RUNsays what each side changed, by column. A newcolumnscolumn (HTTP:columns) gives{here, there, both}for each row both sides changed, and a conflict's detail says whatUSING OURS/THEIRSwould discard againstBY COLUMNSwith that side (both changed age; only here: name; only the parent: email. USING OURS overwrites the parent's age, email, BY COLUMNS USING OURS only age; ...). Settled rows explain themselves the same way, instead ofboth sides changed this row.SHOW MERGE OUTCOMEScounts a merge's conflict with Postgres as a conflict, not a stale read. In an attached folder, a merge refused because Postgres changed its rows was counted understale_reads, and so were refusals because Postgres was busy (a lock timeout) or a world's scope changed meanwhile. Postgres conflicts count asconflictsnow, and the others aren't counted.- Attach: a refresh takes columns added in Postgres. A migration that adds a column (
ALTER TABLE leads ADD COLUMN status text NOT NULL DEFAULT 'new') no longer stops a table's refresh and every merge into it until you attach again: the nextchronos refresh(orserve --refresh-every, or HTTPrefresh) adds the column to main's table with Postgres's type and default, then main takes its values, and says so (added). Merges into the table are refused only until that refresh. A world forked before the column existed leaves Postgres's values in it alone for rows it changes, even settledUSING OURS. Dropped, renamed or retyped columns still need a new attach. - Attach: a tenant's read of a big table no longer holds every merge. A world that checks its reads and ran
select ... where org_id = 42 and status = 'open'on an attached table of more than 100,000 rows was always held for a person ("read leads more broadly than can be checked against Postgres"), as the check read the table whole from Postgres. The check now asks Postgres only for the rows the scan's conditions can match, at the fork and now, so the app writing other tenants' rows doesn't hold it, and a matching row changed or added there still does. A scan through an index keeps its conditions as a scan without one does, rather than the index entries it read, inside Chronos too: a row its other conditions rule out (status = 'won') no longer holds it when it changes. - Attach: a row a world inserts after Postgres added columns gets Postgres's defaults. A world forked before a column was added in Postgres sent it as the default Chronos worked out when the column came over (or NULL, for one added bare):
now()was that moment's,random()NULL. The write-back now leaves the columns the world never had out of its INSERT, so Postgres fills them as for its own rows, and main takes the row back as Postgres has it. - Hourly limits count per tenant for agents held to one.
rows_per_hour,deletes_per_hourandeffects_per_hourcounted each agent's merges on their own, so a product that makes an agent per run (CREATE AGENT run_123 WITH (tenant = 'org_id = 42', valid_for = '1 hour'), as the agent gateway does) started the hour again with every run. An agent madeWITH (tenant = ...)now shares one count with every agent made for that tenant; a merge over it waits for a person, sayingmerges for tenant "org_id" = 42 deleted 100 in the last hour. Agents with no tenant are counted alone, as before. - Every query has a memory limit. Only an agent with
max_memory_mbhad one, so a runaway join by anyone else held rows until the operating system killed the server and every session with it (BranchBench's S4 took one to 55 GB). Now any statement pastCHRONOS_QUERY_MEM(default a quarter of the machine's memory,0for none) fails alone with 53200, and statements together are held to twice that. A hash join's probe checks the limit, andstatement_timeoutand cancels, as it makes rows, on every core: a 120 s timeout used to land at 160 s or later. A join that held 51 GB within 13 s now fails in a second with the server up. An agent'scount(*)over a join bigger than itsmax_memory_mbis now refused too: it holds the joined rows while counting. - Attach: merges keep their connection to Postgres. Every merge into main of an attached folder connected to Postgres anew, with a TLS handshake and a SCRAM login: 4 to 5 of a small merge's 13 or so round trips, so a 1-row merge from Mumbai to Neon in Singapore (70 ms away) took 0.9 to 1 s. The folder now keeps one connection between merges, and refreshes, dry runs,
CHECK READSand the ids reserved from Postgres's sequences use it too. Each use still starts its own transaction with its settingsSET LOCAL, and the connection is kept only when no transaction is left open. One the server closed meanwhile (a restart, a pooler,pg_terminate_backend) fails at the merge's first statement, before anything is written, and the merge connects again once; a connection lost after that is the merge's error, as before. A newCHRONOS_SOURCE_URL(orCHRONOS_SOURCE_FILE) is used from the next merge on, and the connection closes after 5 minutes unused, so a Neon or Supabase database can still scale to zero. - COPY FREEZE is refused once the COPY has begun, as Postgres refuses it. A
COPY ... FROM STDIN FREEZEinto a table the transaction didn't create or truncate (or a partitioned one) was refused before the CopyInResponse; Postgres refuses it after, so the client sends its data, which is dropped. With the refusal first, psql read a script's data lines after the COPY as statements. Found by Postgres'scopy2regression file, which the nightly regress run now runs with psql 16.14, the version of its files. - Both sides adding columns to one table merge. A world and its parent that each changed a table's columns (one adds
tier, the otherloyalty) were refused withboth sides changed table t itself, whatever the columns: merge one side, redo the other. They now merge column by column: each side's new and dropped columns, the world's after the parent's, every row read through the merged table (the other side's new columns null or their default), in a merge, a rebase and a dry run. The same column changed by both, or two columns of one name, is still a conflict. Found by BranchBench's S4, where a dev world moves its column out of the way of the spine's (add, copy, drop) and its rebase was still refused. - A hash join holds only the columns it reads of its other side. That side's rows kept a value's room for every column of their table, the ones nothing reads after the table's own conditions included (as nulls), so a wide table went to disk (
CHRONOS_WORK_MEM) long before its rows needed to. CH-benCHmark's Q3 and Q18 read 4 of order_line's 10 columns: at five warehouses its 1.5 million rows went to disk, and the queries took 3.0 s and 5.3 s; held narrow they stay in memory and take 0.7 s and 1.4 s, as withCHRONOS_WORK_MEM=4GBbefore. A join that does go to disk writes the narrow rows too. Found by BranchBench, where these two queries took 73% of S4's TPC-C spine. - A merge says how many rows both sides changed, and how it settled them.
MERGE WORLDandREBASE WORLDthat settle rows both sides changed (USING,BY COLUMNS,RESOLVE) send one NOTICE with the counts:3 rows both sides changed: 1 combined by columns, 0 settled as x has them, 1 as main has them, 1 as RESOLVE gave them. A client that counts conflicts used to run aDRY RUNfirst and then merge again; the count now comes with the merge. Found by BranchBench, whose merges count conflicts. DIFF ... SUMMARYcounts a diff without reading it.DIFF WORLD a [TO b] SUMMARYgives one row per table and kind of change (table,change,rows), and the changes to tables and other objects asDIFFlists them, without turning any row into JSON. A client that wanted only the counts of a 10,000-row diff used to read every row (0.5 s). Found by BranchBench, whosediffcounts rows.UNDO DIFF a TO breverts a change in one statement. In the session's world it puts back what turnedaintob(two worlds, or one world at two moments, asDIFF WORLD a TO blists it), indexes rebuilt and constraints checked as in a merge. A row changed since is named in a 40001 and nothing is undone, as forUNDO AGENT;SKIP CHANGEDleaves those, andEXCEPT TABLES (log)leaves tables alone. A revert used to beDIFF ... AS SQL, its statements then run one by one: 2 s for 10,000 rows, and a row changed since was overwritten without a word. Found by BranchBench, whoserevertundoes a commit.- LangGraph keeps its state in a world, and pytest gets one per test.
clients/python/chronos_langgraph.pyhasChronosSaver, a LangGraph checkpointer, andChronosStore, its long-term memory store, both on a branch:builder.compile(checkpointer=ChronosSaver(world), store=ChronosStore(world)). A run's checkpoints sit beside the rows it wrote, so the diff a reviewer reads shows both,world.merge()lands them together andworld.discard()drops both, and forking the world forks the thread. LangGraph's own checkpointer conformance suite passes (66 tests;pruneanddelete_for_runsaren't offered). The store'ssearch(query=...)is Chronos's full-text search, typos allowed; it has no vector index.clients/python/chronos_pytest.pyaddschronos_world, a fork of main per test, discarded after it (pytest -p chronos_pytest). Both run in the drivers build (cloudbuild/drivers.yaml). LangGraph's persistence keys threads bythread_idwith nothing per run; this was audit item A5.
The macOS builds need macOS 13 or later, as for 0.2.0.