Skip to main content

The spine and what replicates

The lattice page says of your machine's history: "It isn't replicated or gossiped — it's yours, in one place." For envelope contents, coins, pages and vault secrets that remains exactly right. But it is not the whole picture, and taken as a general statement it is wrong: something does travel between the eighteen nodes, continuously, and this page says precisely what.

Two chains in one table​

Every WORM event is one row in worm_ledger, and every row is linked into two hash chains:

ChainColumnsCoversLeaves the node?
localprev_hash → hashevery row on this nodeno
federatedfed_prev_hash → fed_hashonly network-shareable rowsyes

The local chain is this node's own integrity record. The federated sub-chain is the spine — the only thing that replicates. A row that is not federated has fed_hash = NULL and can never be streamed to a peer, because the streaming query filters on exactly that:

SELECT seq, event_type, owner_zid, zeqond, payload, fed_prev_hash, fed_hash
FROM worm_ledger
WHERE federated AND fed_hash IS NOT NULL AND seq > $after
ORDER BY seq ASC LIMIT $cap

What actually travels​

Always federated — network-level state:

  • issuance — one row per window, the coin (the issuer);
  • pool_inflow / pool_outflow — the treasury pool's flows, which is why snapshot.poolBalance is one number across all eighteen nodes;
  • machine_genesis — a machine's birth: ZID, slug, genesis zeqond, home node, display name, purpose, parent origin;
  • precision_daily — the daily precision summary seal;
  • anchor, tombstone, kernel — cross-node attestation and kernel pinning.

Private by default — per-user value movement:

const PER_USER_ECONOMY = new Set(["mint", "transfer", "burn", "xfer_debit", "xfer_credit", "wallet_delta"]);

These are federated only if the originating machine has opted in (state_machines.network_member = true). No machine context in the payload means private — the code fails private, not open.

Never on the wire at all. The spine carries events, not tables. Your envelope and coin rows (tally_tokens), your machines (state_machines), your pages (state_machine_pages) and your vault secrets (ZSC) are per-node tables. No replication path exists for them, and a node that is down takes them with it. That is the honest limit of "one machine": see shared versus per-node.

You can watch the spine move without any credentials:

curl -s https://zeq.me/api/worm/replication \
| python3 -c "
import sys,json
d=json.load(sys.stdin)
print('asked at zeqond', d['zeqond'], '·', len(d['origins']), 'origins')
for o in sorted(d['origins'], key=lambda o: -o['applied_count'])[:5]:
    print(f\"  {o['origin_node_id']:<28} applied {o['applied_count']:>7}  updated_z {o['updated_zeqond']}\")
"

Each row is one peer's cursor on this node: how far into that origin's federated sub-chain we have verified and applied. On a 2026-09-03 sample zeq.me held 17 origins, cursors ranging from applied_count 7 up to applied_count 152,742 for machine-zeqenvelope#fed.

The receiver recomputes; it never trusts​

applyFederatedRows() does not store what a peer sends. It re-derives it. For every row in a batch, in order:

  1. Linkage. The row's prev_hash must equal the receiver's running chain tip for that origin. Any gap, reorder or omission breaks the link.
  2. Content. The receiver recomputes the hash from the row's own material — sha256(prev_hash | event_type | owner_zid | zeqond | canonical_payload) — using the identical recipe the sender used to write fed_hash. If the recomputation does not equal the claimed hash, the row is a forgery or a corruption.
const recomputed = replRowHash(String(r.prev_hash),
  { event_type: r.event_type, owner_zid: r.owner_zid, zeqond: r.zeqond, payload: r.payload });
if (recomputed !== String(r.hash)) {
  return { ok: false, applied: 0, last_seq: cur.last_seq, last_hash: cur.last_hash,
           first_bad_seq: seq, reason: "fed content hash mismatch" };
}

Failure is all-or-nothing: applied: 0, the cursor does not move, and the response names first_bad_seq. A peer cannot get one bad row past the check by burying it in a good batch.

The same rule applies to this node's own fold. wormRoot() deliberately folds a recomputed content digest of each row rather than the stored hash column — because an attacker with database access could mutate a payload and leave the stored hash intact. Folding the recomputation means any change to a row's material flips the root:

curl -s https://zeq.me/api/worm/root
# {"ok":true,"root":"0c2186cd00cb9d4d…e49b","count":623,"tip_zeqond":2301712918,…}

curl -s https://zeq.me/api/worm/verify
# {"ok":true,"count":623,"first_bad_seq":null,"worm_mode":"off","zeqond":2301712921}

This is convergence by determinism and content addressing, not by agreement. There is no vote anywhere in the path.

Dedup by origin and sequence​

Verified rows land in replica_ledger, keyed by the pair that makes them unique:

INSERT INTO replica_ledger (origin_node_id, origin_seq, event_type, owner_zid, payload,
                            zeqond, prev_hash, hash, applied_zeqond)
VALUES (…)
ON CONFLICT (origin_node_id, origin_seq) DO NOTHING

Three layers of idempotency stack up:

  • Cursor skip. Any row at or below the origin's current last_seq is skipped before it is even hashed, so re-pulling from zero is free.
  • (origin_node_id, origin_seq). Applying the same batch twice inserts nothing the second time. The federated cursor is namespaced `<origin>#fed` so full-ledger and federated replication of the same peer never collide.
  • DISTINCT ON (fed_hash). The balance and pool projections fold the union of this node's own federated rows and rows replicated from peers, deduplicated by content hash — so a row that travels out to a peer and comes home again through a third node collapses to one.

Sequence numbers are not required to be contiguous. worm_ledger.seq is a global BIGSERIAL that can start above 1 and skip on rolled-back inserts, so contiguity would produce false alarms. The hash chain is the real and sufficient guarantee.

The same "determinism plus a unique key" pattern is what collapses duplicate births: a machine's machine_genesis is unique on its slug, readCanonicalGenesis() takes the earliest such event across worm_ledger ∪ replica_ledger, and two nodes that both record a birth converge on one canonical row without ever talking about it.

What the spine is used for today​

Be concrete about this, because the spine is doing less than the design documents imply.

  1. The failover watermark. readSpineLease().tipWindow is the maximum issuance window on the spine — worm_ledger ∪ replica_ledger, federated rows only. A node that takes over issuance resumes from that tip, never from its own ledger.
  2. The issuer-lease store. The epoch and issuer stamped on each seal live nowhere else. The lease is not a coordination service or a lock; it is a fold over rows every node already has. (Both of these are behind ZEQ_FAILOVER=1, which is off on the live eighteen.)
  3. The pool and network genesis. poolBalance = 436469 and network_genesis_zeqond = 2301090211, one distinct value each across all eighteen, are projections of federated rows.
  4. Canonical machine identity. Earliest-birth-wins over machine_genesis.

What it is not yet: a shared database. GET /api/worm/status on the eighteen returns eighteen different ledger_root values and ledger_count between 7 and 153,689. That is expected — the local chain is per-node — but it means claims of the form "every node holds the identical log" are not true today, and the checks below will tell you so rather than hide it.

Tamper-evidence, and the honest state of the quorum​

curl -s https://zeq.me/api/worm/quorum | python3 -m json.tool | head -30

quorumView() asks, for each member X, what every other member has recorded as X's anchor, and flags X as divergent only when an observer's record is at the same height but a different root — a genuine contradiction, not merely a different ledger size.

Two limits worth knowing before you read a quorum_ok: true:

  • The roster comes from the static ZEQ_MESH_PEERS environment list, not the live registry. On zeq.me today that is member_count: 4 — machine-zeqme, machine-zeqproof, machine-zeqenvelope, machine-hulyafield — not eighteen.
  • GET /api/worm/anchors on that node currently returns anchors: [], and every member's observers array is empty. With nothing observed, nothing can contradict, so today's quorum_ok: true is a pass over an empty set — not a positive cross-attestation.

Governance mode is a one-way ratchet, off < evident < enforced, enforced by a database trigger as well as in application code; a downgrade is rejected with code: "RATCHET". Live sweep on 2026-09-03: sixteen nodes at off, two (hulyafield.com, machine.zeq.digital) at evident.

Transport 1 — ZeqSSH over raw TCP​

Node-to-node replication does not go over HTTP. It rides a raw TCP session on the mesh port — tcp_port: 18870 on every one of the eighteen registry entries — with no nginx in the path.

  • Framing. Every message is length-prefixed: a 4-byte big-endian length followed by JSON, with a 1 MiB ceiling per frame.
  • Wire. HELLO (client) → HELLOACK (server) → FRAME / ACK, the last two carrying AES-256-GCM ciphertext.
  • Record keys. deriveRecordKey() derives a per-zeqond AES-256-GCM key by HKDF-SHA256, rekeyed every zeqond and separated by direction, so a captured frame's key never opens a later one.
  • Authentication. The HELLO carries an HMAC-SHA256 over its canonical fields, and the per-session secret is HKDF(ZEQ_MESH_SECRET, sessionId, nonce) — both ends derive bit-identical keys with no public-key exchange. A HELLO more than 300 zeqonds (~4 minutes) out of step is refused.
The TCP transport is not Ed25519 today

ZEQ-DISTRIBUTED-DB.md describes ZeqSSH as carrying "our Ed25519 identity". The shipped lib/zeqTcp.ts does not: it authenticates on the shared ZEQ_MESH_SECRET, and its own comment says "Ed25519 peer identity can layer on later; the record layer is unchanged." The TCP channel is the trust boundary for anything that rides it — which is why a mesh_announce over TCP carries no additional signature. Ed25519 is real and is used on the HTTP peer protocol below.

Peer discovery rides the same channel. announceSelfOnce() sends a mesh_announce with this node's descriptor to every known peer and merges back their peer sets, so the mesh becomes fully connected from a single bootstrap peer with no environment surgery. That registry is what GET /api/mesh/peers serves.

Transport 2 — the signed HTTP peer protocol​

Some reads have to be node-to-node over HTTPS rather than the replication channel — the home-node directory is the clearest example. Those use a separate, signature-based protocol implemented once in lib/peerAuth.ts.

Three headers on every request:

HeaderContents
x-zeq-peer-originthe caller's public origin, lowercased, no trailing slash
x-zeq-peer-zeqondthe caller's zeqond at signing time
x-zeq-peer-sigEd25519 signature over a canonical string

The canonical string is built by the same function on both sides and is scoped to the endpoint and its subject — for the presence read it is `peer-presence|<ZID>|<zeqond>`. Both ends build it from the claimed zeqond in the header, so there is nothing to negotiate.

Three checks on the receiving side, in order:

  1. the origin must be a framework-owned domain (an SSRF and spoofing guard) — otherwise 403 not_a_peer;
  2. the claimed zeqond must be within PEER_ZEQOND_WINDOW = 300 zeqonds (~4 minutes) of ours — otherwise 403 stale_peer_request;
  3. the signature must verify against the caller's published key, fetched from the caller's own GET /api/node/status → publicKeyHex and cached for 60 seconds — otherwise 403 bad_peer_sig (or 403 peer_pubkey_unavailable if the key cannot be fetched).

There is no shared secret in this path. The trust anchor is the signature against a published key, never the transport, which is why the fan-out can be redirected to an in-container port without weakening anything.

# every node publishes the key it will be checked against
curl -s https://zeqproof.com/api/node/status \
| python3 -c "import sys,json;d=json.load(sys.stdin);print(d['nodeId'], d['publicKeyHex'])"
# unsigned, so it is refused — this is the gate working
curl -s https://zeq.me/api/identity/peer/presence/ZEQTEST
# {"ok":false,"error":"not_a_peer"}

A dead peer degrades the answer, never the request. fanOutToPeers() asks every peer the same signed question in parallel and keeps whatever arrives inside the timeout; a peer that is slow or down is reported as unreachable rather than turning the caller's read into an error. That rule holds for every mesh read in the codebase.