What mnestic adds

mnestic is a maintained fork of CozoDB, continued under MPL-2.0 from upstream commit 481af05 (the last upstream commit, 2024-12-04). The query language (CozoScript) and engine semantics are unchanged unless noted here — what changed is performance, correctness, and the primitives an agentic-memory workload actually needs.

Lineage

The fork point is CozoDB main HEAD — 30 commits ahead of the published 0.7.6 crate, which means adopting mnestic already gives you several unreleased upstream fixes for free (including the stored_prefix_join correctness fix). The version was bumped to 0.8.0 to mark the fork's identity rather than 0.7.7.

Highlights by release

0.13.1 — the count rewrite ships on

A patch release. The factorized count() rewrite, opt-in since 0.10.5 while the planner-regression suite was built out, is now on by default: an eligible single-clause count()-over-a-positive-join is counted without materializing the join. Answers are identical by construction — the pass fires only on a provably exact decomposition and leaves declined queries byte-identical — and set_query_factorization(false) restores the previous behaviour. Measured on LSQB sf0.1 (SQLite, release), both paths returning LDBC's published count: q1 72.4 s → 1.05 s (~69×), q6 42.1 s → 0.31 s (~134×).

Shipped alongside it: 0.13.0's expected-token parse hints, which pest 2.8 had silently disabled for anyone resolving dependencies fresh (tracking became opt-in; our caret requirement admitted 2.8; the committed lockfile meant CI never saw it). The floor is now pest = "2.8", the tracking is enabled only around a re-parse of an already-failed script — it is a process-wide global with a real cost, and an embedded engine should not leave it on inside its host — and a weekly fresh-resolve CI lane now covers the class.

0.13.0 — the correctness union, and the features it earned

A nine-bug hardening pass over the storage and index paths, paired with the primitives an agentic-memory workload needs. Ships with mnestic-rocks 0.1.10.

• The RocksDB table options you configure now reach RocksDB. open_db loaded your BlockBasedTableOptions (block cache, block size, cache_index_and_filter_blocks) and then silently threw them away, resetting the table factory to a default-constructed one — so an embedded engine ran with an 8 MB read cache and a 4 KB block size regardless of what its host asked for. Fixed in mnestic-rocks 0.1.10. Every read-path benchmark taken before this measured a slower engine than mnestic actually is. Inherited from upstream Cozo.
• Budgeted hybrid retrieval in one call. hybrid_search gains a budgeted-expansion mode — a graph leg with max_nodes runs BudgetedTraversal's cheapest-first weighted expansion over a cached graph: projection, seeded from the vector/FTS legs' own top-k — and its vector_index/fts_index are now optional, so any non-empty subset of legs can be fused. A payload missing its leg is a loud error, not a dropped signal.
• A datetime standard library (dt_*) with a typed validity bridge. Component extractors, dt_trunc, calendar-aware dt_add/dt_diff, strftime dt_format, and dt_to_validity — which converts float Unix seconds to a Validity where the unit is known. @ and :as_of now accept a Validity-typed expression, closing the seconds-vs-microseconds trap 0.12.2 opened the door on.
• The != inclusion–exclusion count rewrite is restored, behind a type gate (default OFF). Sound this time — it fires only when both inequality operands are variant-stable stored columns of one agreed type, so join equality and op_neq cannot disagree. LSQB q6: 41.7 s → 0.30 s (~140×), count exactly the published oracle.
• Errors that name what the engine expected. Parse errors point their caret at the deepest position reached and list the tokens that would have parsed; index-search diagnostics carry the failing index kind's own code; and import_from_backup refuses a schema mismatch — the one path that could write a value violating its column's declared type.

→ Hybrid retrieval · Datetime functions · Release history

0.12.2 — a float in a validity is now an error

Validity and transaction-time stamps are integer microseconds since the Unix epoch; now() and parse_timestamp() return float seconds. The engine coerced one into the other silently, in all four places a timestamp can enter a validity — the @ selector, the @ (tt: …) / :as_of selector, the validity(...) constructor, and the write path. A fact meant for 2024 was denominated a million times too small and stored at 1970: the row reads back correctly on an ordinary query and is wrong only under time travel, which is exactly where a bitemporal engine has to be trustworthy. On the read side the misread landed before any row was asserted, so the query returned zero rows and no error. All four now reject a float with a message that names the unit and the fix (to_int(<expr> * 1000000)).

Like the 0.12.1 bugs below, this one is inherited — and it predates bitemporality. Three of the four sites are verbatim upstream code at the fork point (481af05, 2024-12-04): the @ selector (expr2vld_spec), the validity(...) constructor (op_validity), and — worst — the write path itself, whose DataValue::List arm of ColType::Validity is byte-identical to upstream's. So is the accessor all three funnel through, Num::get_int, which coerces any whole-numbered float to an i64. Only the transaction-time selector is ours, and 0.10.0 did not introduce the coercion there, it inherited it: Validity columns and @ time travel are a Cozo feature that predates the fork by years. Every CozoDB database with a Validity column has this bug, and the upstream code that writes 1970 still ships.

The idiom that triggers it — Validity default [floor(now()), true] — had exactly one caller we could find anywhere: upstream's own HNSW test, which we inherited unchanged, which had been writing 1970 into upstream's own valid-time axis for as long as the test existed, and which never asserted on the value. See Release history for the upgrade action, and Time travel for what the selector accepts.

0.12.1 — six inherited bugs, found and named

Every bug in this release is inherited from upstream CozoDB and has been latent since before the fork point — not one is a regression the fork introduced. They surfaced in a line-by-line audit of the engine we inherited. Naming them is the point: a dormant project ships none of these fixes, because nobody is reading the code.

• ::reindex <relation> — the repair path. Rebuild a relation's HNSW / FTS / LSH indexes in place from the index configuration the database already stores. It replaces the "drop and recreate the index" advice the bulk-load paths used to give, which meant reconstructing the original ::hnsw/::fts creation script (extractor, tokenizer, filters, ef_construction, m_neighbours…) by hand. Each index is rebuilt from its own stored manifest — an LSH manifest keeps the derived band geometry but not the weights behind it, so a hand-rolled recreate silently changes the index's recall profile.
• Full-text postings leaked on in-place updates (affects every release through 0.12.0). A relation carrying only an FTS index never deleted a row's old postings on a :put over an existing key: terms the document no longer contained kept matching it, the index grew without bound, and BM25 statistics drifted — a measured 55% score error. Existing indexes are already affected; the fix stops new leakage but cannot evict what was written, so affected relations need one ::reindex. LSH does not leak.
• The transaction API reported success for a failed commit. MultiTransaction::commit() discarded the commit's own result, so a failed commit returned Ok — and the HTTP /transact endpoint answered 200 {"ok": true} for transactions that never committed. Change-feed subscribers likewise received events for rows that were never committed.
• Two non-default backends fixed. newrocksdb never armed conflict validation, so concurrent updates were silently lost; sled's del() never deleted (upstream #306). Both now run a transaction-contract suite in CI.
• import_from_backup silently stranded HNSW/FTS/LSH indexes — a restored backup returned nothing from hybrid retrieval, with no signal. It now warns, and both bulk-load warnings point at ::reindex.

→ System ops · Proximity search · Release history

0.12.0 — budgeted weighted traversal

• BudgetedTraversal — the context-fill primitive. A new graph-algo fixed rule: cheapest-first expansion from a set of seeds, over non-negative edge weights, under a required global budget of distinct nodes (max_nodes), emitting the budget's-worth of cheapest admissible nodes as (node, cost, parent, depth). Deterministic by construction — admission follows a (cost, node) total order, and positional edges vs. a cached graph: projection produce byte-for-byte identical output. max_cost bounds admissible path cost; max_depth is an exact hop bound (layered labels, never depth-pruned Dijkstra). An optional gate relation plus an admit: predicate filters mid-expansion — a gated-out node spends no budget and never bridges — and the loop honors :timeout / ::kill. Weights are consumed as costs; monotone transforms like -ln(weight) are the caller's. Measured at the release's merge gate: one call over a cached projection runs 2–4× faster than the production host-side BFS it replaces.
• The optional rayon dependency is bounded >=1.10, <1.11 — rayon 1.11 breaks graph_builder 0.4.x (the CSR-builder crate behind graph-algo), so a fresh downstream resolve now lands on a working pair.

→ Utilities & algorithms · Graph projections

0.11.1 — built-in skyline aggregates

• pareto_min / pareto_max keep the whole Pareto frontier, per group. Given a numeric vector per row, each keeps the non-dominated points under componentwise order (pareto_min = smaller is better on every component, pareto_max = larger), one row per survivor — a query can surface a contested set of equally-good answers instead of collapsing to a single min or max. Mixed objectives (minimize price, maximize quality) use the sign-flip idiom: negate the maximized components and use pareto_min.
• Native dominance — no host registration. Unlike the registered antichain bounded-meet from 0.10.1, the componentwise comparison is built in, so these need no register_bounded_meet_aggr and are reachable from every binding — the PyPI wheel, cozo-bin, langchain, llama-index — through plain run_script. They compose in recursive rules too, and a malformed operand (non-list, non-numeric or NaN component, empty vector) is a loud error.

→ Skyline aggregates · Aggregations

0.11.0 — cached graph projections

• Name a graph once and reuse it across queries. ::graph create g { edges: knows, nodes: person } builds a named, in-memory adjacency over stored relations that twelve graph algorithms reuse via a graph: option, instead of scanning the edges and rebuilding the CSR on every call. On a 400,000-edge graph, reusing the projection took ConnectedComponents from 127 ms to 7.9 ms (16×), PageRank at 20 iterations from 150 ms to 10 ms (15×), and ClusteringCoefficients from 169 ms to 56 ms (3×) — what is cached is the setup, so the gain shrinks as the kernel dominates. The projection is always fresh — it never serves a transaction data that differs from its own scan of the sources, and writing to a source frees what was built from it; under write churn it degrades to build-per-query, never stale. Projections are in-memory and not persisted, and a 512 MiB LRU ceiling (settable from Rust and Python; 0 disables caching) bounds their footprint.
• BREAKING (results): PageRank's default iterations is now 20, up from 10 — a below-upstream default that is measurably non-convergent at the default epsilon. Pass iterations: 10 to restore the old numbers. PageRank also now accepts an optional node relation (as ConnectedComponents already did), so edge-less vertices are ranked instead of silently dropped, and warns when the iteration cap stops it short of epsilon.
• Fix: an empty edge relation no longer aborts the process in seven graph algorithms (it used to panic; they now return no rows, as PageRank already did). And multi_transaction no longer deadlocks a process by parking a rayon worker for the transaction's whole lifetime — it now runs on a dedicated thread, which affects every caller of that API.

→ Graph projections · Utilities & algorithms · System ops

0.10.7 — a join-reorder plan fix, and factorization from Python

• Fix: the greedy join reorder no longer demotes a full-composite-key filter to a partial-key expansion. A tie-break bug (full_key_lookup_bonus) could pull a high-fan-out edge ahead of a more selective atom and regress a cyclic-join query (a benchmarker measured LDBC-SNB LSQB Q3 go from ~19s to a timeout; the fix restores it). Result sets are unchanged and the min-new-vars speed-up from 0.10.5 is preserved. Separately, the Python binding now exposes db.set_query_factorization(True) / db.query_factorization(), so the 0.10.5 factorized-count() kill switch — previously Rust-only — is toggleable from Python (default still off).

0.10.6 — legacy databases open again after upgrade

• Fix: relation catalogs written before 0.10.0 no longer fail to open. An urgent upgrade-safety patch. On 0.10.0–0.10.5 a database created before 0.10.0 (or last updated by an index/rename/destroy path) could fail to open with "Cannot deserialize relation metadata from bytes" — the bitemporality work added a RelationHandle field mid-struct, which broke positional decoding of the older relation-catalog layout and could take the whole database down. 0.10.6 makes catalog decoding tolerant of the legacy layout and switches every catalog write to a self-describing, field-named encoding so it can't recur. No migration — legacy databases open as-is and re-canonicalize on their next write. Anyone who upgraded a pre-0.10.0 database to any of 0.10.0–0.10.5 should upgrade. Two internal items ride along: the deterministic join-reorder pass is refactored to a pure function with no behavior change, and the storage-rocksdb Python-wheel CI is hardened.

0.10.5 — interruptible queries, wall-clock budgets & a stat-free join reorder

• ::kill and :timeout now actually interrupt a running query. ::running and ::kill dispatch before opening a storage transaction, so a ::kill no longer queues behind the very query it is trying to stop (the old failure on the mem/SQLite backends). The per-query poison flag is now checked every 4096 pulls inside the relational-algebra enumeration, so even a long single-rule join that emits no rows is finally interruptible. The Python close() no longer raises "Already borrowed" against a live run_script.
• Per-query wall-clock budget — set a deadline three ways: the in-script :timeout <secs> option, a per-call run_script_with_options, or a Db-wide set_default_query_timeout. The effective deadline is the minimum of whichever are set — a :timeout can only tighten the budget, never extend past the default. Expiry raises a distinct eval::timeout error (a ::kill raises eval::killed); Python run_script gains a timeout= kwarg. Wasm carries no wall-clock budget (no monotonic clock there).
• Deterministic greedy join reorder — default on — a stat-free, min-new-vars reorder of a conjunction's positive relation atoms, removing the N³ blow-up a naively-ordered query (the kind an LLM authors) falls into (measured 54.5× on the repro; N³ → N²). Results are unchanged — it is the identity on any already-greedy order, and it is not a cost-based optimizer. Opt out per query with :reorder written; a residual Cartesian step is warned and annotated in ::explain.
• Automatic factorized count() rewrite — opt-in, default off — behind set_query_factorization, an eligible single-clause count()-over-join is rewritten into per-key counting sub-rules that compute a bit-identical (exact-i64, Int-typed) answer without materializing the join (the benchmark measured 4–342× versus a factorizing optimizer). It fires only on shapes it can prove exact; an always-on detector adds a factorization advisory to ::explain.
• RocksDB now ships in the PyPI mnestic wheel — CozoDbPy("rocksdb", path) works straight from pip install mnestic (was compact/SQLite-only). The sdist stays compact, so the persistent engine is wheel-only.
• Bulk import_relations into an index-bearing relation now warns — the bulk path maintains B-tree indexes but not HNSW/FTS/LSH, so imported rows stay invisible to vector/text search until the index is rebuilt; a warning now flags it.

→ Interruptibility & timeouts · Join reorder · Factorized counting

0.10.1 — antichain aggregate & interval primitives

• Dominance bounded-meet — the antichain / skyline aggregate — register_bounded_meet_aggr opens the bounded-meet category to a host-supplied strict partial order, keeping the non-dominated (Pareto-frontier) set of operands per group, one output row each. max_survivors is a mandatory resource guard — overflow is a loud error, never a silent truncation. Rust-embedded only in v1 (host closures do not cross the Python/served surfaces).
• Interval primitives — interval_overlaps(a, b) builtin and the interval_coalesce(span) aggregate over half-open [start, end) list intervals. Touching spans do not overlap but do coalesce ([0,5) + [5,10) = [0,10)); empty spans overlap nothing; malformed spans are loud errors, never silent falses.
• Correctness fix — the bit_and/bit_or meet aggregates now report whether the value actually changed, so a non-changing fold no longer re-enters the semi-naive delta every epoch (the same defect family as the 0.10.0 and/or changed-bit fix).

→ Skyline aggregates

0.10.0 — bitemporality & provenance semirings

• Bitemporality — engine-assigned transaction time alongside Cozo's valid time. A crash-safe monotone commit clock stamps every write to a TxTime relation; reads default to the current belief and time-travel with an @ selector or the :as_of query option; ::history / ::history_gc (persisted floor) / ::evict (audited hard deletion) manage the record's lifecycle. Answers "what did we believe at time T about period Y" in-engine, with current-belief reads within ~4–12% of the single-axis baseline.
• Provenance semirings — the same recursive rules can compute existence, cost, confidence, or evidence. register_custom_aggr admits user-defined absorptive combines into recursion; min_cost_k is a bounded-meet aggregate returning the k best derivations per answer with the evidence chains that justify them; and :reconcile is recompute-based belief revision that keeps derived annotations consistent under base-fact retraction, composed with the tt axis.
• Four upstream bugs fixed along the way — the inverted changed-bit in and/or meet aggregates, a panic on negated validity atoms, wrong answers from prefix-truncated temporal-column joins, and the braced-%return imperative parse panic.

→ Bitemporality · Provenance semirings

0.9.0 — read-only Cypher & corrupt-DB tooling

• Read-only Cypher query surface (alpha, opt-in cypher feature, off by default) — translate a subset of openCypher to CozoScript so the engine can be evaluated without first learning Datalog. New API: run_cypher / cypher_to_script, driven by a caller-supplied schema mapping the property-graph model onto stored relations. Datalog stays the native, full-power language; this is a read-only on-ramp (no write clauses). The published PyPI wheel ships without it.
• ::repair_corrupt <relation> — surgically deletes truncated tuples (short value bytes from interrupted writes) by their intact store keys, giving applications an alternative to dropping a database that fails integrity checks.
• Non-panicking ::index create — a single truncated tuple used to panic the whole index build (and could make a database unopenable); corrupt tuples are now skipped with a loud error naming the relation, index, and arity mismatch. Both were banked as 0.8.6 and first published here.

0.8.5 — flat parallel HNSW builds & snapshot reads

• Flat in-RAM parallel HNSW bulk build — ::hnsw create now builds the graph in contiguous integer-indexed memory (the hnswlib/pgvector/Lucene layout) and inserts in parallel with per-node locks, eliminating the tuple encode/decode and allocator traffic that dominated the old temp-store build. MNESTIC_INDEX_BUILD_THREADS controls worker count.
• Plain-snapshot read path for read-only scripts (RocksDB) — immutable scripts no longer open a pessimistic transaction; they read the base DB through a plain MVCC snapshot with no lock-manager bookkeeping (keyed point read p50 28.5 → 23.9 µs, −16%, measured).
• Batched HNSW neighbour reads — the search path fetches unvisited neighbours' vectors through one RocksDB MultiGet per expansion step instead of a serial point-get per neighbour; the win case is cold-cache / larger-than-RAM data.

0.8.4 — per-leg retrieval detail & an FTS concurrency fix

• Per-leg retrieval detail — ReciprocalRankFusion(..., detailed: true) and HybridSearch::detailed switch the output to the long format [item, fused_score, list_id, leg_rank, leg_score], one row per contributing list — the mechanism behind a "why was this retrieved" surface, reconstructing the fused score exactly. Exposed to the Python binding as detailed: True.
• FTS avgdl concurrency fix — 0.8.3's durable doc-stats counter was one shared storage key rewritten inside every document transaction, making concurrent writers to an FTS-indexed relation conflict on a single row lock (and losing updates through an unlocked read-modify-write). The counter is now process-cached and scan-seeded, with no shared storage key in the hot path; per-query avgdl stays O(1).

0.8.3 — native 3-way fused recall & BM25 full-text

• Native 3-way fused recall — hybrid_search now takes typed GraphLegs, so graph proximity fuses with the vector and keyword legs in one call. Each leg expands from seeds over a stored edge relation up to max_hops and ranks reached nodes by minimum hop distance. The fused call runs all three signals at ~41.55 ms p50, roughly 4× faster than hand-decomposing it into separate queries.
• BM25-correct full-text search — the default ::fts scorer is now Okapi BM25 (term-frequency saturation + document-length normalization), OR sums per-term contributions, and avgdl is an O(1) durable counter rather than a per-query scan. This lifted fused recall from 0.75 to 0.954. tf and tf_idf remain selectable for byte-identical upstream scoring.

→ Hybrid retrieval (RRF + MMR + graph legs)

0.8.2 — non-blocking HNSW index builds

Building an HNSW index no longer holds the base relation's write lock for the duration of the build, so concurrent reads no longer stall for minutes. The graph is built off-lock under a snapshot and bulk-published with RocksDB's SstFileWriter / IngestExternalFile.

→ Non-blocking HNSW builds

0.8.1 — one-call hybrid retrieval & faster builds

• hybrid_search runs HNSW + FTS (+ optional graph traversal), fuses with Reciprocal Rank Fusion, and optionally diversifies with MMR — in a single typed call, replacing roughly seven hand-written Datalog rules.
• HNSW index builds ~3× faster (20k × 128: 135 s → 43.6 s, measured release), with a byte-identical result.
• The C++/RocksDB bridge is now a maintained fork, mnestic-rocks, keeping the importable name cozorocks.

→ Hybrid retrieval (RRF + MMR)

0.8.0 — fixes & agentic-memory primitives

• Equality pushdown — *rel[k, ..], k == <value> now compiles to a keyed stored_prefix_join instead of a full scan (~28–29× faster single-row primary-key lookups at 5k rows, measured).
• ReciprocalRankFusion / MaximalMarginalRelevance fixed rules (aliases RRF / MMR) — the fusion and diversity primitives behind hybrid retrieval, usable directly in Datalog.
• ULID functions — rand_ulid() and ulid_timestamp() for lexicographically-sortable, time-ordered keys.
• Parser fix — identifiers that begin with a keyword literal (nullable_column, trueValue, falsey) now parse correctly.

→ Equality pushdown · ULID identifiers

Not affiliated with CozoDB

mnestic is not the official CozoDB and is not affiliated with or endorsed by its authors. All credit for the original design belongs to Ziyang Hu and the Cozo Project Authors. See License & attribution.