Skip to main content

The issuer and failover

One coin per zeqond, for the whole network. Not one per node — one, full stop. This page is how that holds when there are eighteen nodes, and what happens when the node doing the sealing stops.

Exactly one node seals​

sealIssuance() (shared/api-core/src/lib/worm.ts) begins with a single guard:

if (process.env.ZEQ_MESH_ISSUER === "false") {
  return { sealed: false, window: 0, from_window: 0, elapsed: 0, amount: "0" };
}

ZEQ_MESH_ISSUER is set to false on every node except the one designated issuer. Those nodes compute nothing, seal nothing and mint nothing; they replicate the issuer's rows and project the same balances from them. That is the mechanism — and the measurement below shows it holding on the live fleet.

Alongside it, nodeIssuer.ts provisions one canonical machine per node — slug node, purpose: "master", owner node, with tally_supply.mint_rate_per_zeqond = 1. Every other machine on the network carries rate 0. A user machine, a spun-up machine, a node's own mini-machine: all zero. The node row is the only one with a nonzero rate, and only on the node that is actually the issuer does that row ever get to seal.

You can watch the single writer from outside. GET /api/worm/replication returns this node's replication cursor per origin — one row per peer, keyed <node-id>#fed:

curl -s https://zeq.me/api/worm/replication \
| python3 -c "
import sys,json
d=json.load(sys.stdin)
print('zeqond', d['zeqond'])
for o in sorted(d['origins'], key=lambda o:-o['last_seq'])[:4]:
    print(' ', o['origin_node_id'], 'last_seq', o['last_seq'], 'updated_z', o['updated_zeqond'])
"

Sampled twice, 52 zeqonds apart (2301712548 → 2301712600), exactly one of the seventeen cursors moved:

machine-zeqenvelope#fed   last_seq 153827 → 153841   updated_z 2301712544 → 2301712597
machine-hulyasorg#fed     last_seq  23412 (frozen)   updated_z 2301665992
machine-zeqproof#fed      last_seq    176 (frozen)   updated_z 2301665890
machine-zeq-dev#fed       last_seq     89 (frozen)   updated_z 2301665992
...and twelve more at last_seq 7, all frozen at zeqond ~2301665992

The operations reports name machine-zeqenvelope as the configured issuer, and the live cursors agree: it is the only origin producing federated rows.

The exactly-once window key​

Issuance is keyed by the shared network zeqond, not by any node's private clock:

export const ISSUANCE_WINDOW = 1;
export function windowFor(z, window = ISSUANCE_WINDOW) { return Math.floor(z / Math.max(1, Math.floor(window))); }
export function issuanceEventKey(z, window = ISSUANCE_WINDOW) {
  return `zeq-issuance-v1:w${windowFor(z, window)}`;
}

With ISSUANCE_WINDOW = 1 the window index is the zeqond, so window 2301711961 has key zeq-issuance-v1:w2301711961 — the same string on every node, at the same moment, forever:

curl -s "https://zeqsdk.com/api/mesh/issuance/at?z=2301711961" \
| python3 -c "import sys,json;d=json.load(sys.stdin);print(d['key'], d['amount'])"
# zeq-issuance-v1:w2301711961 0.999999999999

That key is enforced by a partial-unique index on payload->>'key' in worm_ledger. If two nodes ever seal the same window — during a takeover, on a merge, by mistake — the second insert comes back as Postgres 23505 and is swallowed:

if (code === "23505" || /duplicate key|unique/i.test(msg)) {
  // Another node already sealed this window (replicated in). Exactly-once holds.
  return { sealed: false, duplicate: true, window: gz, from_window: fromWindow, elapsed, amount };
}

This is the whole exactly-once mechanism. There is no proposer, no vote, no quorum. Two honest nodes cannot disagree about a pure function of the tick, so a vote would prove nothing; a unique key over a deterministic string does the job instead.

Everything below here is behind a flag​

The rest of this page describes ZEQ_FAILOVER=1. On the live eighteen the flag is unset, which means sealIssuance() runs the pre-failover path byte for byte — the issuer's own ledger as watermark, a cap of 8 windows per seal, no lease stamp — and runFailoverElectionOnce() returns on its first line. If the issuer stops today, issuance stops until an operator intervenes. The code is shipped and tested; it is not switched on.

The issuer lease​

Under the flag, every sealed row carries two extra payload fields. Here is the shape, built around the first takeover row from the drill below — window, from_window, elapsed, epoch and issuer are the drill's logged values; the rest follow from them by the shared arithmetic:

{ "key": "zeq-issuance-v1:w2301676684", "window": 2301676684, "from_window": 2301676606,
  "elapsed": 8, "rate": 0.999999999999, "amount": "7.999999999992",
  "phase": 0.9999813435332383, "genesis_pegged": true,
  "epoch": 1, "issuer": "drill-b" }

epoch and issuer are the lease: a claim, written into the append-only log itself, that says who the writer is and which generation of writer they are. readSpineLease() folds the spine back into one lease — highest epoch wins; within an epoch the holder is whoever stamped the highest window; tipWindow is the global maximum window regardless of epoch. Because it is a fold over rows every node has, every node derives the same lease.

A node may seal only under sealDecision():

SituationResult
ZEQ_MESH_ISSUER=falsenever seals (replica)
just won an election, promotedEpoch > lease.epochseals, stamping promotedEpoch
spine carries no holder yet (legacy rows only)the configured issuer adopts the lease at epoch 0
lease.issuer === selfseals at the lease epoch — steady state
another node holds an epoch at or above oursdemote: set ZEQ_MESH_ISSUER=false, seal nothing

That last row is what closes the dual-writer window. There is no moment where two nodes both believe they are the issuer and both write.

The election​

electIssuer() in failoverSpine.ts is a pure function — no database, no network, no environment reads — so every node computes the same answer from the same spine.

The field is self ∪ the live mesh_peers registry (listPeers()), optionally narrowed by ZEQ_FAILOVER_ELIGIBLE. It is not the static ZEQ_MESH_PEERS environment list; that was one of the six defects the audit found.

The rank is the canonical spine tip — which is identical on every node by construction — with lowest node id as the tiebreak:

return list.sort((a, b) => (a < b ? -1 : a > b ? 1 : 0));

A node's own ledger row count is deliberately not an input. Under a "longest chain" rule machine-hulyasorg, which carries 23,412 legacy rows, would win every election forever regardless of whether it had ever issued anything.

Promotion needs three conditions at once:

  1. the spine tip has not advanced for more than ZEQ_FAILOVER_SILENCE_Z windows — default 60 Z, about 47 seconds at 0.777 s per zeqond;
  2. the current lease holder is unreachable from here (an anchor probe over the mesh transport returns nothing), or there is no recorded holder;
  3. this node is the deterministic successor — the first ranked candidate that is reachable and is not the holder being replaced.

Everything else holds. A node that can still reach the holder never promotes. A node whose spine is still advancing — because it can see the tip through any other peer — never promotes, which is the split-brain guard for a partitioned minority.

Three cold-start guards exist because a staged drill found the code promoting every node at once on a fresh network:

  • an empty spine (tipWindow <= 0) never elects — silence on an empty spine is infinite, so only the configured issuer may bootstrap epoch 0 (reason: spine_empty_bootstrap_only);
  • a field of one never promotes — a node whose registry has not filled in yet cannot claim to have won anything (reason: field_of_one);
  • no election runs during the boot grace window, ZEQ_FAILOVER_BOOT_GRACE_Z, default 40 Z (~31 s), so a freshly restarted process pulls the spine before it trusts its view of the lease.

The election itself is throttled to ZEQ_FAILOVER_CHECK_Z, default every 16 zeqonds (~12.4 s).

Demotion when the original issuer returns​

The returning node does not need to be told anything. It reads the spine, sees a lease at an epoch at or above the one it last claimed held by someone else, and stands down — first in the election loop:

[failover] DEMOTED → replica (spine lease held by another node at ≥ our epoch)

and again, independently, at the seal, in case the seal path is reached first:

[failover] DEMOTED at seal — spine carries a higher/foreign issuer lease (no dual-seal)

Both paths set ZEQ_MESH_ISSUER=false, which sealIssuance() re-reads on every call.

The takeover watermark, and the gap​

A promoted node resumes from max(its own ledger tip, lease.tipWindow) — never below the shared spine tip. That is what stops a fresh replica with an empty ledger from re-sealing from window 0 and colliding with every key already on the spine.

How much it issues on that first seal is a policy dial:

  • ZEQ_MAX_ELAPSED_PER_SEAL, default 8, caps a single seal. Under the default, an outage of T windows permanently under-issues T − 8 windows. At roughly one ZEQ per zeqond that is about 4,630 ZEQ for a one-hour outage, never minted.
  • ZEQ_FAILOVER_CATCHUP_MAX_Z, default 0 (off), lets the first seal after a gap cover more.

Catch-up is one-shot, and the tests prove it: the row is keyed window = gz, so the tip jumps to the current window on that first seal and any remainder above the cap is forfeited — recorded as window − from_window − elapsed and therefore auditable, never quietly issued later. Neither setting can inflate: elapsed ≤ gap always.

The drill: measured, not projected​

The failover path was exercised in an isolated three-node drill on the VPS on 2026-09-03 — its own compose project, its own bridge network, three throwaway Postgres/Redis/app stacks bound to 127.0.0.1 only, with ZEQ_FAILOVER=1, SILENCE_Z=60, BOOT_GRACE_Z=40, CHECK_Z=16 and CATCHUP_MAX_Z=0. The live eighteen were not touched — no environment change, no restarts. The harness was torn down completely afterwards. Result: PASS.

MeasuredValue
Unit tests23 / 23 green (failover.test.ts)
Issuer A killed03:02:07Z, last own window 2301676622, 45 rows, Σ 184 Z
B promoted03:02:49Z — 42 s after the kill, silence 74 Z, epoch 0 → 1
B's first takeover rowwindow 2301676684, from_window 2301676606, elapsed 8, epoch 1, issuer drill-b — resumed from the shared tip, not from 0
C's decisionheld, reason: successor_is_drill-b — same successor computed independently
Windows sealed below the tip0 on all three nodes
Duplicate issuance keys0 on all three (rows == distinct keys: 85 / 86 / 86)
B ≡ Ccommon-prefix md5 identical (2cde7d98…8ea2); C lagged by ≤ 1 pull cycle (20 Z)
A restarted03:03:16Z
A demoted03:03:48Z — 32 s after return; 0 seals after return, no dual-seal window
ConservationΣ elapsed 352 Z (drill-a 184 + drill-b 168) over a 406 Z span; 54 Z under-issued, exactly the documented cap policy — nothing inflated

The drill also changed the code. On its first run, every node promoted itself, because an empty spine reads as infinitely silent and the peer registry had not filled in — which is why the three cold-start guards above exist and have regression tests.

Known limits​

  • Not enabled. ZEQ_FAILOVER is unset on all eighteen nodes. Today the issuer is a single point of failure for issuance.
  • Replication lag can double-count. If the dying issuer's last rows had not replicated, and catch-up is on, the successor can cover windows the old issuer had already sealed — up to one fed-pull cycle (20 Z) — when those rows merge back. With the default cap this is harmless.
  • Failover covers issuance only. It does not move envelopes, coins, machines or pages. Those stay on the node that made them; see what is per-node.
  • The design documents disagree with the code. MESH-ISSUANCE.md and DECENTRALIZED-MESH-PLAN.md describe a leaderless mesh where every node mints the deterministic coin and duplicates collapse on merge — "no elected issuer to lose". The determinism that would make that work is shipped and verifiable (the issuance/at check above). The leaderless mode is not: the shipping code is single-writer by an explicit owner decision, and the planned retirement of the node issuer has not happened.