Skip to main content

WORM governance & cross-node quorum

Your machine stays node-local: its live state, chain and clock live on the node where it was spun up (see Nodes and node-locality). WORM is a second, independent layer on top of that — a write-once-read-many ledger that folds to a single root hash and is cross-checked across the fleet.

Two things on this page have changed since it was first written, and both matter:

  • the quorum roster is still the static ZEQ_MESH_PEERS bootstrap seed, so member_count reads a handful — 4 on zeq.me, 2 on zeqproof.com and zeqenvelope.com — not eighteen. Widening it to the live mesh field was tried on 2026-09-03 and reverted the same day: see the note below;
  • WORM is no longer purely advisory. The federated sub-chain of worm_ledger — the spine — is the failover watermark and the issuer-lease store.

What WORM stores​

WORM keeps its own tables, separate from the live machine:

TableHolds
worm_ledgerthe append-only, write-once event log — the source of both hash chains
worm_balancesevent-sourced balances projected from the ledger
worm_projection_metaprojection watermark for rebuilds
worm_tombstonesowner-signed, equation-proof deletions
replica_ledgerverified rows replicated from peer nodes
node_anchorsthe last-seen ledger root of every peer this node checks

Every row is linked into two chains: a local one (prev_hash → hash) covering every row on this node, and a federated one (fed_prev_hash → fed_hash) covering only network-shareable rows. Only the federated sub-chain leaves the node. See the spine and replication.

The ledger is folded to one 32-byte root. Because it is write-once, that root can only change by appending — never by rewriting — so one hash detects tampering anywhere in history. The fold deliberately recomputes each row's content digest rather than trusting the stored hash column, so an attacker with database access cannot mutate a payload and leave the root intact.

Read the surface — three curls​

curl -s https://zeq.me/api/worm/status    # governance mode, ledger count, folded root
curl -s https://zeq.me/api/worm/root      # the folded root + the tombstone root
curl -s https://zeq.me/api/worm/verify    # walk the chain and report the first bad row

Live on 2026-09-03:

{"ok":true,"worm_mode":"off","kernel_match":true,"ledger_count":648,
 "ledger_root":"63dbe7542ff9…","zeqond":2301714829}

{"ok":true,"root":"df3aece9653b6069c68880a7a8dfc03f1ec60810c58c56c47460d3e3102f4da8",
 "count":654,"tip_zeqond":2301715246,
 "tombstones":{"root":"0000…0000","count":0,"tip_zeqond":null}}

{"ok":true,"count":654,"first_bad_seq":null,"worm_mode":"off","zeqond":2301715264}

ledger_count differs by node and moves between calls — that is the local chain doing its job, not an inconsistency. Across the eighteen on 2026-09-03 it ran from 7 (most nodes) to 154,388 on the issuer zeqenvelope.com, with eighteen different roots. kernel_match: true on all eighteen: the pinned kernel hash 7804d842d5f5… over 1,612 operators equals the current one.

The quorum — a real sample​

curl -s https://zeq.me/api/worm/quorum | python3 -m json.tool
{
  "ok": true,
  "self": "machine-zeqme",
  "member_count": 4,
  "quorum_ok": true,
  "flagged": [],
  "mesh": [
    { "node": "machine-zeqme",
      "live_root": "3b504d33556613b92008431af161701fe4cc1bb6dfd925e82a2cd27b7a3f71a2",
      "live_count": 643, "reachable": true, "divergent": false,
      "contradiction_count": 0, "observers": [] },
    { "node": "machine-zeqproof",
      "live_root": "c90c60510ed2a751e63132648f5b62a9f9bef390d85a35a43abcc02314c9b1e0",
      "live_count": 186, "reachable": true, "divergent": false,
      "contradiction_count": 0, "observers": [] },
    { "node": "machine-zeqenvelope",
      "live_root": "70936c841b7e594f87774a6de99a784a351554fed9c603499e4318ea5233518b",
      "live_count": 154257, "reachable": true, "divergent": false,
      "contradiction_count": 0, "observers": [] }
  ],
  "zeqond": 2301714829
}

Sweeping the fleet on 2026-09-03: quorum_ok: true and flagged: [] on every node, with member_count between 2 and 4 depending on that node's seed. quorum_ok: true here is a weak statement, for the two reasons below.

Why the roster is not the live mesh, and why that is on purpose for now

Reading the roster from resolveMeshPeers() — the env seed union the live registry, the same field the replication puller and the issuer election use — is obviously more correct, and it was shipped on 2026-09-03. It was reverted the same day, because it destabilised the issuer.

Every member in the roster is probed by probePeerAnchor, which opens a raw TCP socket. Widening from a 3-entry seed to 17 peers multiplied the sockets per quorum call about sixfold. A socket error emitted asynchronously, after its promise has already settled, never reaches the route's try/catch: it surfaces as an uncaughtException, and the ZeqSSH guard deliberately exits the process on any error code outside its known-socket set. On the issuer — the one node whose restart halts issuance mesh-wide — that meant five restarts before the cause was understood.

The fix is not to widen the roster; it is to make the probe path escape-proof first (bounded concurrency, an explicit error handler per socket, and the swallowed-code set audited), and only then widen it, with its own drill. A correct number on a dashboard is not worth a crash-looping issuer. Until that lands, read member_count as "how many peers this node's seed lists", and use GET /api/mesh/peers for the real roster of eighteen.

What quorum_ok: true actually proves​

Tamper-evidence, not agreement. Nothing here is a vote. No node's root has to match any other node's root — they are different ledgers and they should differ. quorumView() asks a narrower question: for each member X, what has every other member recorded as X's anchor, and does any of those records contradict what X is publishing right now?

A contradiction has a precise definition:

const sameHeight   = rec.ledger_count === lv.count && lv.reachable;
const contradicts  = sameHeight && rec.ledger_root !== lv.root;

Same height, different root — one node cannot have had two different histories at the same length. A different height is not a contradiction, just a stale observation. divergent is true when a reachable member has at least one contradicting observer; quorum_ok is flagged.length === 0.

Be honest about how strong today's pass is​

It is weak. Not wrong — weak, and for a reason you can measure yourself:

curl -s https://zeq.me/api/worm/anchors
# {"ok":true,"self":"machine-zeqme","anchors":[],"zeqond":2301714925}

Sweeping the quorum on all eighteen nodes on 2026-09-03 turned up one single observation in the entire mesh — machine-zeqdigital holding an anchor of machine-zeqcash at ledger_count 1 while zeqcash is live at 7:

{ "by": "machine-zeqdigital",
  "root": "4bfcb08fdadde94eeb69d9b6a053458074d594b83218668a96c5d282800baba5",
  "count": 1, "same_height": false, "contradicts": false }

Different height, so it cannot contradict anything. Every other observers array in the fleet is [].

With nothing observed, nothing can contradict. Today's quorum_ok: true is therefore a pass over an almost-empty set — it says "no node has caught another node rewriting history," which is true, but no node has recorded enough of another node's history to catch it. Read it as the mechanism is wired and reporting the right roster, not as eighteen nodes have cross-attested each other.

The unambiguous check available today is the single-node one: GET /api/worm/verify walks a node's own chain and returns first_bad_seq. That is real, and on 2026-09-03 it returned null with count: 654 on zeq.me.

Governance mode is a one-way ratchet​

off < evident < enforced, enforced by a database trigger as well as in application code; a downgrade is rejected with code: "RATCHET". Live sweep on 2026-09-03: sixteen nodes at off, two — hulyafield.com and machine.zeq.digital — at evident. GET /api/worm/network-mode reports "mode":"networked" on zeq.me.

WORM has never frozen a live machine or blocked a write in production, and evident mode does not gate anything either.

Where WORM is no longer advisory​

The line "advisory and never gates" is now too broad. The federated sub-chain carries two things the issuance path depends on:

  1. The failover watermark. readSpineLease().tipWindow is the maximum issuance window on the spine (worm_ledger ∪ replica_ledger, federated rows only). A node taking over issuance resumes from that tip, never from its own ledger — which is what stops a fresh replica re-sealing from window 0.
  2. The issuer lease. The epoch and issuer stamped on each seal live nowhere else. The lease is not a lock or a coordination service; it is a fold over rows every node already has, so every node derives the same answer.

Both are behind ZEQ_FAILOVER=1, which is unset on all eighteen live nodes — so today they are shipped and tested code, not running behaviour. See the issuer and failover.

The full WORM endpoint set​

EndpointReturns
GET /api/worm/statusgovernance mode + ledger count + folded root + kernel match
GET /api/worm/rootthe folded ledger root + the tombstone root
GET /api/worm/verifychain walk; first_bad_seq is null when clean
GET /api/worm/kernelpinned vs current kernel hash (the frozen math)
GET /api/worm/anchorthis node's signed anchor (what peers pull to cross-check it)
GET /api/worm/anchorsevery peer anchor this node has stored
GET /api/worm/quorummembers, reachability, observers, divergence, quorum_ok
GET /api/worm/replicationthis node's replication cursor per peer origin
GET /api/worm/network-modewhether this node is networked or standalone
GET /api/worm/balancesbalances projected from the ledger
GET /api/worm/tombstones/:owner_zidowner-signed deletions

How nodes cross-check each other​

A node probes each member's live anchor over the mesh transport, stores what it learns in node_anchors, and then asks each other member what it has recorded. A node behind NAT can still take part in the observing half: it reaches out, cross-checks and reports, without needing any inbound reach. That outbound-only property is what makes the empty observer lists above a configuration gap rather than a design limit.

Relationship to node-locality​

This does not change node-locality. Your live machine is still deterministic and local — a contract fire still doesn't wait for consensus. WORM sits beside it: a write-once spine any observer can fold to a hash and compare. Node-local for speed and a single sealed history; cross-node quorum for tamper-evidence — as soon as the nodes are actually observing each other.