YMRTECH — Notes

Running a 27B model on a 3090: a budget, not a flag

2026-10-16T00:00:00-04:00

Fitting a 27B-parameter model on a single consumer card is an arithmetic exercise, not a configuration one. There are two unknowns: weight memory at the quantisation you choose, and KV cache at the context length you actually want. Everything else — offload counts, thread counts, model-loading flags — is downstream of those two.

The failure mode is rarely “it does not fit”. It is “it fits at 4k context and dies at 32k”, or “it fits, and it generates at two tokens per second”. Both are budget failures, and both are visible before you serve a single request if you measure instead of guess.

This is a retired experiment. The GPU box’s local model stack has been switched off — as of 2026-09-16 the serving endpoint had no listener at all (TCP refused) and the host’s AI services were disabled. What remains useful is the method and the harness, not the deployment.

The two numbers that matter

Throughput claims about local inference reduce to two measurements: tokens per second (TPS) and time to first token (TTFT). Both must be taken at the context length you intend to use. A headline TPS on a 512-token prompt is not evidence about a 32k-token session; it is evidence about a different workload.

The retired baseline illustrates this because it is one point, not a verdict: roughly 24 tokens/second decode on a short prompt with a TTFT of 0.36 s, for a 27B model on a 24 GB RTX 3090 with the length ceiling configured at 90000 tokens. That 24 is a point on a curve. Decode TPS falls as context grows, because the KV cache competes for the memory the weights occupy — and that curve, not the number, is the interesting output.

One trap worth keeping: reasoning models stream their thinking in delta.reasoning_content, not delta.content. A TPS counter that reads only content reports zero tokens while the model works perfectly — it looks like a broken server and is a broken counter. Ask the server for exact usage with "stream_options": {"include_usage": true} so the final chunk carries the true prompt-token count and the prefix-cache hits.

Weight memory is a budget line, not a setting

Quantisation trades memory for quality and speed. It is a budget line, not a preference — and it cannot be chosen independently of the context you need, because both draw on the same pool. On the 24 GB card this ran on, the 27B model at the quantisation in use stayed resident at roughly 21 GB of VRAM. That leaves a few gigabytes for the KV cache and compute buffers; a larger context comes out of that remainder.

There is a version of this that looks like a flag. The arch pin is real: RTX 3090 is compute capability 8.6, and pinning CUDA architectures to 8.6 rather than building for every architecture is the single biggest build-time lever. It also changes the derivation hash of every CUDA package, so the official caches stop matching and packages like nccl, onnxruntime and triton recompile from source — hours on a small core count. That is a cost you budget, not a switch you flip.

The term people forget

KV cache grows with context length and with concurrent sessions. It is the term missing from most “will it fit” arithmetic, because it is invisible until the second request arrives. A configured ceiling of 90000 tokens does not mean a 90k-token session is affordable once the resident weights and the cache share one card; it means the server will accept it and then decide.

The budget is a line, not a table:

VRAM >= weights(quantisation) + KV(context x concurrent sessions) + overhead

Every term on the right is knowable before launch: the 21 GB resident figure is the first, and the context you need decides the second.

Measure the curve, keep the harness

A single result decays — a driver update, a requantisation, a firmware change moves it. A benchmark you keep and re-run is worth more, because the output you care about is the curve across context lengths. This fleet’s standard harness grows a simulated agent session across roughly twelve turns to about 88k input tokens — near the 90000 ceiling — then runs needle-recall probes, then replaces the middle of the history with a lossy summary and re-probes to measure recall fidelity after compression. Content is deterministic (seeded), so runs are comparable. A --target flag and a quick smoke mode cover shorter runs.

The conclusion is narrow. On a single desktop card, the useful question is which context lengths your workload actually needs — because that answer, not the parameter count, determines what you can run at all. The first full run of the harness has not been completed and published; what exists today is the method and the file, which is what makes the next run comparable to it.

What you can check

Run the same harness at two context lengths and look at the TPS/TTFT curve for yourself. One number is an anecdote; two points on the same curve are a measurement.
Record the quantisation alongside every number. A TPS figure without the quantisation that produced it is not reproducible, because the quantisation sets the first budget term.

The infrastructure these services ran on — hosts, GPU modules, the deploy health gate and its rollback — is described at how the stack is built. The benchmark above is the same discipline as the audit page’s signed artifacts: measure, keep the artifact, publish the method, and let the curve, not the headline, be the claim.

Cookieless A/B testing without lying to anyone

2026-10-15T00:00:00-04:00

A/B testing has a reputation it earned. Most of the tooling sold for it answers “which variant won” by first answering who are you, and keeping the answer — a cookie, a session identifier, a device graph — for as long as someone wants to keep asking. None of that is a requirement of measurement; it is a requirement of a business model that got bundled with measurement.

You can assign a visitor to a variant, count the outcome, and hold no identifier for them — by deriving the assignment from something coarse and rotating instead of storing it, and by being clear up front about which questions that makes unanswerable.

The assignment is derived, not stored

The design this site’s stack prescribes, and what its privacy policy describes in §2.1, is: assign server-side — never a client-side splitter, never a blocking third-party script. The edge service decides which variant a request sees; the page only reports what happened. Nothing is set on the client at all.

The assignment key is not an identity. It is an HMAC over a coarse input — a client address reduced to a network-sized prefix, plus the coarse browser family — under a key that rotates on a schedule. Two consequences follow directly:

Within a rotation window the assignment is stable and reproducible from the request alone, so nothing is remembered between the impression and the conversion.
Across the boundary the same person hashes to a different key. The two windows cannot be joined back together — by us, or by anyone who later obtains the logs.

That is the mechanism, and it is also the trade. Hash the coarse UA family alongside the address, because one NAT or office IP would otherwise pin everyone behind it to a single arm. Decide the day-bucket timezone once and document it — UTC avoids the DST cliff where a bucket boundary silently moves.

Attribution does not depend on the key. The variant rides in the links: through the CTA, into the signup entry, validated against the declared variant set, into the payment provider’s session metadata, and out as one structured conversion line from the payment webhook. The authoritative conversion is that event — never a click, never a platform-reported number.

What the rotation costs

The cost is precise, and it belongs in the design document before the test rather than in the readout afterwards: you cannot follow a returning visitor across the rotation boundary. Retention-style metrics — did the person who saw arm B come back ten days later — are out of scope by construction. A visitor-level split on conversions also needs volume to resolve anything: below roughly fifty visits a day it cannot detect a sane effect size, and the move there is to test at content or campaign level and aim the instrumentation at clicks, scroll and dwell.

The second cost is caching. Per-visitor content served under the site’s default cache header means one visitor’s variant is served to everyone, so the response must carry Cache-Control: no-store, private plus Vary on the header the proxy itself sets. Bots are served the control variant and marked uncounted, so they cannot skew one arm.

A system that cannot state the question

Here is the argument underneath this, and it is not a compliance argument. A measurement system that requires identifying everyone to answer a question is usually a system that has not defined the question. This design forces the question into the open — what is the key, how long is it stable, what can we not measure — because the answers have to be written down before the first impression. The questions it refuses are not sacrifices; they are the ones nobody had specified.

The same discipline shows up elsewhere here. The fair-share shaper that caps the VPN tiers keeps no per-peer traffic record, because the no-log posture forbids it, so fairness comes out of the data path instead: a guaranteed share plus a burst ceiling, read from nothing but the live counters the kernel holds in memory. The constraint did not weaken the mechanism. It chose it.

What you can check

The recorded and not-recorded lists. The privacy policy §2.1 states exactly what a page measurement contains — that a page was viewed, the A/B variant when a test is running, the browser’s own timing, uncaught JavaScript errors — and what it does not: no cookie, nothing in local or session storage, no IP stored with a measurement, no cross-site identifier.
The retention numbers. Log records are kept 14 days, and the page-timing measurements travel inside those same records — there is no separate metrics store for them, so they expire with the logs. The aggregate byte counters are kept at most 90 days. Every period is enforced by the servers and visible in the public configuration.
The build gate. The measurement library is pinned by hash in the public repository, and every build re-checks that the served file matches the pin, that the page sets no cookie and leaves nothing in browser storage, and that it contacts nothing but this domain’s own collector.
The checklist for any proposed test: what is the assignment key, how long is it stable, what can we not measure, and what gets published.

None of this claims cookieless measurement is strictly better. It claims the questions a test cannot answer should be decided before the test, not discovered in a dashboard afterwards — the same reason the audit page is built from signed artifacts rather than a status badge. A claim you cannot inspect is only a claim about intentions.

A local DKIM check beats every mail-a-service validator

2026-10-14T00:00:00-04:00

The instinct when mail looks wrong is to send a message to a validator and read the page it prints. It is a reasonable first move and a weak last one. A third-party validator tests one message it received: its own path, its own envelope, its own timing. Your signing path — your selector, your submission route, your relay — is what you have to be able to vouch for, and the check that does that runs on your own host.

The path you actually ship

On this fleet, submission is not SMTP from a client to the internet. It is mail.ymrtech.com:587 STARTTLS over the WireGuard mesh, with SASL per mailbox — so the address that sends and the address that authenticates are two different facts, and both matter. Port 25 does not leave the host (587, 2525, 465 and 80 reach from it; 25 does not), so external delivery goes through a smarthost with a sender-scoped relay map rather than a global relayhost.

That matters for verification because the relay is part of the path under test. Whatever you measure has to be measured through the pair you actually ship: this mailbox, through this submission port, out through this relay.

Your own host already stamps the verdict

Third-party validators exist partly because the receiving side records what it checked. Your receiving side can do that too, and hosts/mail/default.nix says how:

# rspamd stamps NOTHING for local/authenticated mail ... a purely internal
# submission carries no Authentication-Results ... This host vouches for
# the submission path instead. The stamp lives in the submission-only
# cleanup service, and reaching that service already requires SASL over the
# wg0-only firewall rule below ... Inbound mail from the internet (25) is
# untouched and still gets rspamd's real verdict.

So there are two verdicts to read, and they are not interchangeable. Inbound mail from outside carries the receiving server’s own Authentication-Results — real verdicts, and note that such a pass rides on relaxed DKIM alignment, since a VERP envelope like bounce.<opaque>@return.<domain> cannot satisfy aspf=s. Mail handed in over submission is vouched for by a submission-only stamp recording a true smtp.mailfrom per identity, derived from the account config rather than hand-listed, so adding an alias cannot silently produce mail that looks unauthenticated.

One caveat, because it wastes sessions: a message delivered to a mailbox on the same server never passes through a receiving MTA’s authentication check. Its headers show DKIM-Signature and Received, and no Authentication-Results. When someone asks you to send them mail “so I can inspect the headers”, say plainly that a local copy carries no verdict and point them at an external mailbox’s “Show original” — while you verify the signature itself, which needs nobody’s cooperation.

Read the result with the selector attached

A bare dkim=pass is a weak observation. Record the signer domain, the selector and the body-hash result together, because they fail independently:

The signing config pins domain and selector: rspamadm configdump dkim_signing shows the path and the selector, and the rspamd log prints DKIM_SIGNED{ymrtech.com:s=mail}.
A selector mismatch is what a dkim=pass without a selector hides, and it is exactly what a rotation breaks.
A body-hash mismatch with a valid RSA header signature usually means your verifier’s relaxed canonicalization is wrong, not the sender’s. Get the padding check right and the false negative disappears.
Check SPF against the envelope sender the relay actually used, not the From: header. Here DMARC passes on SPF alignment alone for mail sent from this host (aspf=s, envelope domain ymrtech.com), which means a broken DKIM costs reputation rather than delivery — a good reason never to use “it arrived” as your DKIM test.

Rotation is the outage you are pre-empting

A DKIM selector change is a mail outage waiting to happen unless both the old and the new selector are published before the switch. A local check is the cheapest way to prove both resolve: query the old name and the new one, and confirm the new key is the complete value, not a truncation.

That last failure has a signature. The record is 410 characters with a p= of 392 (an RSA-2048 SPKI), and a record pasted into a DNS user interface is easily cut to the first 255-character chunk — which was the real defect on this domain, not a “stray control byte”, a wrong diagnosis worth remembering. Byte-compare, never eyeball.

What you can check

The submission shape. The mail host comments in hosts/mail/default.nix: the submission stamp, the SASL-only firewall rule, and inbound port 25 keeping rspamd’s own verdict.
The signing selector. rspamadm configdump dkim_signing on the mail host, and the DKIM_SIGNED{...:s=...} symbol in the rspamd log — the two facts a validator’s web page never shows you.
The key bytes. The published TXT against the key file: 410 characters, p= of 392, compared programmatically rather than by eye.

Then send through the production path, read the verdict your own system stamped, and keep the output — it is the artifact you would have to produce anyway. For how the same discipline applies to a claim you cannot inspect, see the audit page and how the fleet is configured.

Treating a prober as a customer

2026-10-13T00:00:00-04:00

Look at an unauthenticated write endpoint — a form that sends mail — and you will usually find one problem being discussed and another being missed. The first is abuse: someone finds the URL and uses your endpoint to send things. The second is measurement: every probe of that endpoint looks, in your own numbers, like a person who wanted to reach you.

The second problem survives fixing the first.

Where the credential lives decides who can use the endpoint

The site is a static Jekyll build, so the POST goes to /contact/submit, which Caddy proxies to a small loopback-only listener that relays over the mesh to the fleet’s mail host. The endpoint requires a shared token, and a deploy step is where it becomes real:

# It is deliberately NOT baked into the site (the repo is public and
# the token must be rotatable without a rebuild): it is written here, at
# deploy time ... If the file is absent the endpoint stays locked and the
# form shows the visitor a mailto fallback rather than failing silently.

That last sentence is the design worth copying. An absent secret is a fail-closed condition: no token is published, the endpoint rejects everything with 403, and the page still offers a working mailto link rather than a form that appears to work and loses the message.

Rejections should be cheap, and ordered

pkgs/contact-form/contact_form.py states its ordering as a design constraint, and the ordering is the argument: method, content type, content length and rate limit are checked before any parsing, and parsing is capped before any SMTP connection is attempted.

wrong content type gives 415
a missing or oversized Content-Length (cap: 16 KiB) gives 413, refused before the body is read
a missing or wrong token gives 403
more than 5 accepted submissions per source address per hour gives 429

The token is a header the page’s JavaScript sets, not a form field, so it is not part of a plain form POST, and it is compared with hmac.compare_digest. One line in the file is the decision most endpoints get backwards:

# Only an authenticated caller consumes rate-limit budget: the limit is
# there to bound a visitor (or a token-scraper), not to let unauthenticated
# probes lock out real submissions.

A rate limit any anonymous request can spend is a denial-of-service primitive aimed at your own visitors; here the cheap rejections never touch the budget.

A form in a comment still passes a naive check

The site also ships a gate, scripts/check-contact-form-visible.sh, and its header names the mistake it exists to catch: a grep for <form passes even when the markup sits inside an HTML comment. So it verifies the way a browser renders — it strips  pairs, then asserts that id="ymrtech-form" and the controls name, email, message and website are all in the visible remainder, and that a stray opener has not swallowed the page.

That is the same lesson as the status claims elsewhere on this site: the audit page is built from signed artifacts for the same reason — check the artifact in the state it is actually in.

The half that survives the fix

Now the measurement half. A 403 from a token-less prober and a 200 from a real visitor are both one line against /contact/submit. If the number you watch is derived from paths, your funnel counts attackers as demand — and unlike the spam, that does not stop when the endpoint gets better.

So:

Rate-limit by behaviour and by source, not by User-Agent. See don’t ban the Googlebot: that header is chosen by the client.
Separate “a request arrived” from “a person tried to reach us” before counting anything. The endpoint’s own rejections are the interesting signal, and here they go to its journal — bad token from <ip>, rate limited <ip> — while the host’s log collector picks up Caddy access logs rather than service stdout. Rejections are legible on the host, not in the log store, until the collector’s glob is extended.
Treat the form’s success metric as untrusted until you can say what was filtered. A conversion count you cannot decompose counts traffic, not people.

The asymmetry: you can reduce spam and clean up your metrics, but a public unauthenticated endpoint will keep receiving some non-human traffic. The useful goal is not zero — it is that none of it contaminates a decision. The same trap is in a test request is a real request: an observation made from the wrong position is evidence about the position.

What you can check

The ordering, in the file. pkgs/contact-form/contact_form.py in the nix-config repository: the 415/413/403/429 sequence and the comment that anonymous probes do not spend rate-limit budget.
The fail-closed deploy. The Publish contact-form token step in ymrtech-deploy.yml warns and publishes an empty token when the host-readable file is unavailable — and the endpoint then stays locked.
The visibility gate. Run scripts/check-contact-form-visible.sh against a built _site/contact/index.html; it names the specific control it cannot find.

For how a missing token and a missing log line are recorded as evidence rather than assumed, see the logging stack and what no logs cannot promise.

Why the key that decrypts everything is not in the backup

2026-10-12T00:00:00-04:00

A backup is a claim about the future: that on the day it matters, the bytes come back. The interesting failures are not disk failures — they are dependency cycles you built yourself and did not notice.

The repository password is a secret

This fleet backs up to Cloudflare R2 with restic: file state every 15 minutes with a 90-second jitter, the database dump daily (since 2026-09-29 in a unit of its own, so it is no longer regenerated inside every backup run). The module is modules/fleet-backup.nix, and its header states the intent in one line:

# Repo passphrase is `restic-passwd` in sops (shared fleet-wide, on purpose:
# any host must be able to restore any other host's repo during recovery).

Two consequences follow immediately. The passphrase is not per-host, so it cannot be per-host-recoverable. And it is a sops secret, so reading it requires an age identity — /var/lib/sops-nix/keys.txt, root-owned 0600, plaintext on disk.

The recovery path is therefore: to read the backup you need the passphrase, and to read the passphrase you need the identity.

Read that as a cycle check

Both ends of that chain are things you could carelessly store inside the backup. The module refuses both:

# NOTE on /var/lib/sops-nix: this dir holds the host's age identity
# (/var/lib/sops-nix/keys.txt, root:root 0600) which decrypts secrets.yaml.
# It is PLAINTEXT on disk, so it is deliberately NOT backed up raw here:
# the restic passphrase is shared fleet-wide, so a raw copy would place an
# unencrypted private key in R2, readable by any host holding that passphrase.

That is the whole argument in four sentences. Because the passphrase is deliberately shared, a raw copy of the identity would be readable by anything that can read the archive — the backup would contain the key to the backup. The path set is /var/lib/acme, /var/lib/caddy, /etc/nix/netrc and /var/lib/hermes, plus a pg_dumpall on Postgres hosts, and /var/lib/sops-nix is not in it. The raw Postgres data directory is excluded too, because it is version-fragile; the dump is authoritative.

The identity cannot live only on the hosts either. Each host generates its own age key on itself and the private half never leaves the machine; sops.age.sshKeyPaths = [ ] on every host, so the ssh host key is no longer convertible into an age identity. A terminated host therefore cannot supply its own key.

Escrow, and each file’s deliberate omission

The answer is in .sops.yaml, as a second family of creation rules:

# Disaster-recovery escrow of each host's age identity. Owner + operator
# only: no host can unwrap another host's key.
- path_regex: secrets/age-escrow/[^/]+\.age$
  key_groups:
  - age: [*admin_truva, *hermes_agent]

One rule, no per-host exclusions to get wrong — and the recipient list got shorter rather than longer, deliberately. The first version of this escrow encrypted each host’s identity to every other host, on the reasoning that the host which dies cannot unwrap its own key, so a surviving peer should be able to. That reasoning builds a cross-host escalation path: any compromised host can unwrap every other host’s identity. The escrow is now readable by the owner and the operator only, and the fleet was re-keyed on 2026-09-23 so that the peer-encrypted ciphertext is retired rather than merely unused — the identities it belongs to exist nowhere in the fleet, and no host is a recipient of any escrow blob.

Note where the escrow lives: in the private configuration repository as sops blobs under secrets/age-escrow/<host>.age, so it is recoverable from git alone, without depending on R2 — the system that would be the problem in the scenario the escrow exists for. The one piece that is genuinely human-held is the owner identity, which decrypts the store itself.

The tension is worth saying out loud rather than smoothing over: escrowing a key moves risk from the archive to wherever the escrow lives. A sops blob is safe only because of the recipients listed in it, and a recipient list of two people means the store’s availability rests on those two people. That location needs the highest protection in the system, and it is not the backup server.

A backup you have never restored

The only claim worth making about a backup is a restore you have actually performed. docs/disaster-recovery.md in nix-config-private is written as a runbook — provision the instance, restore the identity, apply the flake, restore state, rejoin the tunnel, repoint DNS, verify — with a recovery point of 15 minutes for file state and a day for the database dump, and a recovery-time target under an hour for a full host rebuild. It also records what is deliberately not backed up, which is the part most runbooks leave out.

It ends with a block titled Verify this before you need it, and the commands are the point:

sops -d --input-type binary --output-type binary secrets/age-escrow/vpn.age >/dev/null && echo escrow-ok
age-keygen -y <(sops -d --input-type binary --output-type binary secrets/age-escrow/vpn.age)

The printed public key must equal that host’s anchor in .sops.yaml, and it has to be run where the owner or operator identity is present: hosts are no longer recipients of any escrow blob, including their own. That is a two-line test of the step the runbook labels “WITHOUT THIS NOTHING WORKS”, and it is worth running on a schedule — because a runbook nobody has exercised is documentation of an intention, and a backup system’s documentation is a restore runbook plus the date it was last exercised.

As of 2026-09-27 there is a date to quote: repository integrity checked on all three hosts, and a file-level restore performed on each, with the results recorded in the backup and recovery statement. There is now a second date for the harder half. On 2026-09-28 both database hosts were restored from the physical repository and recovered to a chosen point in time — the restore that matters when a record is wrong rather than when a host is gone — and its measured durations are recorded in the same place. What neither drill shows — a full host rebuild — is stated in the same place, next to them.

What you can check

The cycle, in the module. nix-config/modules/fleet-backup.nix: the paths = [ ... ] list, and the comment explaining why /var/lib/sops-nix is absent from it.
The escrow rules. Not publicly checkable: they live in nix-config-private/.sops.yaml, and the blobs under secrets/age-escrow/ are in the same private repository. What the public repository carries is each host’s public identity anchor in .sops.yaml, which is what the verify block compares against.
That the escrow is testable today. The verify block in docs/disaster-recovery.md, run as the owner or the operator.

For the sibling problem — a status you can inspect versus one you have to trust — the green deploy that deployed nothing makes the same argument about a CI badge. The rest of the host configuration is described in how the fleet is built.

Don't ban the Googlebot

2026-10-11T00:00:00-04:00

A catch-all edge that feeds a deception dashboard and CrowdSec is a good place to notice a category error. The rule that gets proposed is almost always the same: block the scanners by User-Agent. It takes a minute to write, it never fires wrongly on your own browser, and it is close to worthless.

The client chooses the string

A User-Agent header is text the client writes. Any scanner can send Googlebot, and the ones that matter usually do, because impersonating a crawler is a cheap way past naive rules. So a User-Agent deny rule filters on a field controlled by the party you are filtering: it blocks the crawlers that identify themselves, and repels nobody who will not simply pick another string.

Which means the block lands on the traffic that pays for your site, not on the traffic you were worried about.

What can be decided, and by whom

There is one fact about a crawler that the client cannot forge: the address it connects from. The publishers document the ranges they originate from, so this fleet reads those documents instead of guessing.

sources = {
  googlebot = {
    url = "https://developers.google.com/search/apis/ipranges/googlebot.json";
    hash = "sha256-n17GcnxmJ4244sglbrP0VzILCXkWbiBJ/ka6nmUOgu8=";
  };
  googleSpecialCrawlers = { url = "https://developers.google.com/search/apis/ipranges/special-crawlers.json"; ... };
  bingbot = { url = "https://www.bing.com/toolbox/bingbot.json"; ... };
};

hosts/common/global/crawler-ranges.nix fetches three manifests and turns each prefixes list into CIDRs. The ranges are not copied into the repository. They are fetched at evaluation time and pinned by SRI hash, and the header says exactly why: a publisher that silently edits its ranges becomes a hash mismatch — a build failure a human reviews — instead of a silent widening, or a silent narrowing, of the fleet’s own defence.

A security control you cannot diff is a control you cannot review. A list of ranges that changes underneath you is not data you own; it is a number you hope is right. Pinning the bytes makes the change visible. It also makes the fetch a fixed-output derivation, so the URL is contacted only on a cold cache or a deliberate hash bump, not on every rebuild.

Fail closed, and where rDNS belongs

Two details turn the module from a list into a control.

If a manifest stops having the documented shape, evaluation aborts with the URL that changed, so the failure reads review this publisher’s new format rather than defend with an empty list. And every derived entry must match [0-9a-fA-F:.]+/[0-9]{1,3}, because CrowdSec parses whitelist entries with netip.ParsePrefix: a bare address makes the whole postoverflow file fail to compile and the agent refuses to start, which took this fleet down once. Catching it at evaluation means the error names the publisher instead of a truncated store hash.

Reverse DNS is deliberately not required on top. These are the publishers’ own ranges, and at BGP level nobody else originates from them, so a forward-confirmed PTR adds no assurance here — it would only lose the whitelist whenever reverse DNS is slow or missing for a legitimate crawler. The PTR check belongs with the publishers who document no ranges at all; CrowdSec’s hub seo-bots-whitelist matches crawlers by PTR suffix, which is the right check for them.

The consumer is a CrowdSec whitelist of its own, fleet/whitelist-verified-crawler-ranges, so a verified crawler never becomes a decision, whatever a scenario says. It is a separate whitelist rather than extra entries on the fleet’s own-infrastructure list because fleet.crowdsec.whitelistCidrs is a listOf option with a non-empty default: assigning it from a second module replaces that default outright and silently drops the mesh range. Every whitelist is evaluated independently, so an extra entry is equivalent and cannot clobber the first.

The asymmetry, and the limits

The cost of a false block is not symmetric. A blocked malicious client changes its User-Agent and is back in a second. A blocked search crawler is invisible to you — no error page, no alert, no mail — and expensive to detect, because the only symptom is traffic that never arrives.

Two limits are worth stating rather than hiding. An address inside a publisher’s range is not a personality: range allow-listing cannot tell two clients behind the same address apart, so it is a statement about origin, not intent. And an allow-list is a standing exemption, so it needs an owner and a review. The procedure attached to this fleet’s allowlist pull requests is exactly that discipline: the acceptable diff is one appended line, and anything unidentifiable escalates to a human instead of being merged.

What you can check

The module, in full. nix-config/hosts/common/global/crawler-ranges.nix. The three hash = values and the throw on an unexpected manifest shape are the whole argument in code.
That the ranges are the publishers’ own. Each url is a manifest you can fetch yourself; the hash only makes a change visible.
That the whitelist is separate. Search the module for s01Whitelist; the entry is named fleet/whitelist-verified-crawler-ranges, not the own-infrastructure list.

The practical rule: filter by behaviour and by verifiable origin, keep the allow-list small, reviewed and diffable, and never ship a User-Agent deny rule as your security story. For the rest of the defence — the bans, the log line, the dashboard — see the logging stack and how the fleet is configured.

A test request is a real request

2026-10-10T00:00:00-04:00

“I tested it and it worked” is one of the most reliable sentences in operations, and one of the least informative. A filtering resolver has no simulation layer: the query you send is a query, the answer is a policy decision, and whether it is the one your users get depends on who you were when you asked.

There is no test mode

A filtering resolver has exactly one execution path. A request to allow a name — here, a client proposing a whitelist entry from the portal — proceeds down it as production traffic would: the submission becomes a pull request on nix-config-private appending one line to modules/adguard/allowlist.txt, that line reaches main, the deploy renders it into the resolvers, and the request mails the address on record.

The review procedure for those PRs says it in one line: the submit path has no test mode — whatever you submit becomes a PR, reaches the fleet policy on main, is deployed, and mails the address on the request. Hence the reserved host, whitelist-test.ymrtech.com, already in the allowlist, instead of a name you would refuse.

An exemption hides the policy from the tester

A filtering resolver applies policy per client. That is a feature: the tier a device runs on decides which lists answer it. It is also the trap. If the client you test from is exempt from the rule you are testing — a different tier, an allowlisted source, a bypass the operator added — your test cannot observe the rule at all. It observes the exemption.

This is not specific to DNS. Any test performed by an actor privileged relative to the production path is evidence about the test, not about production. The operator’s own machine is the classic privileged actor: it sits on a different resolver address, may carry settings no customer has, and is the client an ad-adjacent domain is most likely to have been allowlisted for by hand.

So “the client told me it is broken, I checked and it works” usually means you asked from a position where the answer is different.

The cache is a third party

A resolver cache serves whatever it already holds, and those entries outlive the change you are verifying. This fleet’s loopback resolvers carry cache-min-ttl = 60. AdGuard Home also rewrites its own config in place and reloads its filter lists from the URL declared in the policy file before it serves them, so a retry loop against a resolver mid-reload produces a spurious empty answer that reads as “blocked”.

A name that fails right after a policy change may be failing from cache; a name that resolves may be resolving from cache from before the policy existed. Either way you measured a cache, or an exemption, and called it the policy.

How this class of request should be reviewed

An allowlist entry is a lasting, reviewable change to a security control, so it deserves the scrutiny any policy edit gets. The procedure attached to these PRs asks four things:

Who asked, and why? A vendor portal, a SaaS the client uses, a school or employer host, or a CDN for one of those is plausible. A host that does not resolve, or a reason that does not match it, is not.
What is the blast radius? One line in one shared file changes answers on every instance in the fleet, not just the requester’s device: the lists are declared in one place and deliberately not overridable per host.
Is the block load-bearing? An ad exchange, an analytics or telemetry network, a tracker — the block exists on purpose and stays. Same for credential, auth and banking hosts without clear subscription context: those escalate to a human.
Is the diff exactly what it claims? One appended rule, no reordering, no removals, no second file. Anything else is a red flag before it is a question.

The rule underneath all four: a request is a policy proposal, judged on the host and the reason, never on the fact that someone tested it and it worked.

What you can check

How to tell whether your own test request was privileged — three questions, where any “yes” invalidates the result:

Which client did you ask from? If it was not the device whose experience you care about — a laptop in the same tier, a phone on the same resolver — you tested a neighbour. Ask from the failing client.
Which resolver answered? Print the address your client actually queries. A machine pointed at the operator’s resolver is a different question from a client on the tier in question.
Was caching in play? Ask for a name that cannot already be in any cache, and re-run after the policy settles rather than while the resolver is still loading rules.

Then record the response code: “blocked” and “not found” are different failures, and only a recorded code keeps them apart later. This is the same discipline as the audit page: a check that runs from the client’s position and publishes its own result beats any number of successful probes from somewhere convenient.

For a related case of the same mistake — a green status that measured the test harness — see the green deploy that deployed nothing.

The DNSSEC trap in a zone you answer yourself

2026-10-09T00:00:00-04:00

There is a failure mode that looks like an outage and is not one. A name that resolved for months stops resolving everywhere at the same instant — not for one client, but for every machine that asks. The zone is fine, the records are fine, nothing was deleted. What changed is that somewhere in the answer path a resolver tried to prove the answer, could not, and said so.

DNSSEC is why, and the trap is that the failure is not proportional to the mistake.

SERVFAIL is not NXDOMAIN

A validator that cannot build a chain of trust from the root to the answer does not serve the unsigned answer as a courtesy. It returns SERVFAIL. The point of validation is to refuse answers it cannot check.

# A validating resolver, asked for a name it cannot validate
$ dig +short ymrtech.com
# (nothing — rcode SERVFAIL)

# The same name, asking the resolver to skip validation
$ dig +short +cd ymrtech.com
11.0.0.7

Those two commands are the whole diagnosis. +cd sets the checking disabled bit, so it shows what the resolver could actually find: if +cd answers and the plain query does not, the data is present and validation is what failed. The records were never the problem.

At the client, SERVFAIL is indistinguishable from a broken zone: “the name does not exist” is what you see, and it is wrong. It resolved a minute ago, it resolves from the server; only the validator differs.

A zone you answer yourself

This fleet answers its own public names internally. The list lives in one place, hosts/common/global/internal-dns-zone.nix:

primary — AdGuard Home on 11.0.0.1:53, upstream unbound on :5353,
standby — AdGuard Home on 11.0.0.7:53, upstream unbound on :5335.

Both read the same records, so a client pointed at either address gets the same answers. Without them, a fleet host talking to one of our own services leaves through the provider’s NAT and re-enters with an external source address.

But ymrtech.com is a DNSSEC-signed public zone, and this is where the trap closes: an internal answer carrying a private address has no signature a public validator can accept — there is nothing to check. So the resolver does the only thing it is allowed to do:

It answers SERVFAIL, not the record.

That sentence is the comment on the zone definition, written when the overrides were added. The remedy it prescribes is one option on the validating hosts — in hosts/giga/default.nix:

# The internal ymr zone is unsigned and answered through AdGuard Home, which
# does not forward DNSSEC records. With the root trust anchor loaded (nixpkgs
# default), the validator answered SERVFAIL for every *.ymr name (verified:
# +cd resolves, plain query fails).
domain-insecure = [ "ymr" "ymrtech.com" ];

hosts/mail/default.nix carries the identical line with the identical reasoning, and both cite the same verification: +cd resolved to 11.0.0.7 while the validating query failed. The public host never hits this at all — it serves ymr from local-zone "ymr." static, so the zone never reaches a validator. Three hosts, three relationships to the same records, and only the two that forward needed the declaration.

The trade, not the mistake

It is tempting to file this under “we broke DNS with DNSSEC”. That reading hides the useful part. DNSSEC buys an answer you can trust and charges a hard failure mode for every break in the chain: a record that is stale, unreachable, or answered from a split-horizon view used to degrade softly — a client fell back, an old address kept working. Under validation there is no soft option. A hairpin answer (internal clients served public addresses, or the reverse) is precisely what a validator rejects, so it does not degrade; it disappears.

The trade is worth paying when the trust anchors are agreed everywhere, and it is a trap when they are not.

What you can check

Ask for SERVFAIL specifically, not just NXDOMAIN. When a name that used to resolve stops resolving, run both queries above against the resolver the affected client actually uses. +cd answering where the plain query does not is the signature.
Test from a real client, not from the server. Whether validation succeeds depends on the resolver in the path, so a query from the authoritative side proves nothing about the laptop. Point a client at the resolver it uses in production and repeat.
Look for the declaration before you look for the bug. On this fleet, domain-insecure in hosts/<host>/default.nix decides whether an internal answer survives validation. If a host validates and answers internal names, those two facts need to agree in writing.
Prefer unsigned internal zones unless everything agrees: keep a zone you answer yourself unsigned, or make sure every resolver and every client that validates shares the same trust anchors. Half-signed is the state that produces the outage.

The failure is invisible from the middle: nothing in a zone file, a record list or a service page shows an answer being refused for lack of a signature. That is the same argument behind the audit page: a claim you cannot inspect from the client’s position is only a claim about the server’s intentions.

For the wider resolution picture, see how the stack is built.

The config change that never restarted

2026-10-08T00:00:00-04:00

On a declarative system it is tempting to believe that editing a file is the change. On NixOS it is not. A config file is content, and nothing restarts the process holding the old copy unless the unit says so. The deploy can be green, the file can be correct, and the running system can be unchanged.

The fleet’s own notes have the incident that makes this concrete, and it is worth quoting exactly.

The feature that never appeared

The hermes-agent module renders its settings into config.yaml — but defines no restart trigger. The observed result: config.yaml contained a new platforms.email block while the gateway was still the pre-deploy process (ExecMainStartTimestamp from the previous day, NRestarts=0), and “the feature just never appeared.” The file had the change. The process did not.

That is the whole failure: a config-only deploy that reports success, and a long-lived process still answering from the configuration it loaded a day earlier. EnvironmentFile-style configuration behaves the same way — it is read at start, so an edited file has no effect until the unit restarts.

What NixOS gives you, and what it does not

NixOS offers restartTriggers for exactly this: a unit carrying restartTriggers = [ ... ] restarts when the content of those files changes, which is how a change to a generated config file pulls a restart behind it. Without a trigger, nothing does.

systemd.services.hermes-agent.restartTriggers = [
  (pkgs.writeText "hermes-agent-settings-trigger" (builtins.toJSON {
    settings = config.services.hermes-agent.settings;
    environment = config.services.hermes-agent.environment;
  }))
];

Two details from that wiring matter. The trigger hashes environment as well as settings, because an env-only change would otherwise go stale — and the definition has to be gated on whether the module is enabled, because on a host without it the reference fails evaluation.

The other half is the reload path. A unit with a working ExecReload can pick up a config change without a restart; here unbound is left to reload rather than restart so it keeps its resolver cache, set with systemd.services.unbound.reloadIfChanged = true on the mail and giga hosts. But a unit without a usable reload action will accept a reload request and do nothing useful — a no-op that looks like success. And a graceful reload is not always enough: the notes record that Caddy’s graceful reload does not make it re-read a new user-supplied tls file pair, and only a restart does.

Verify the process, not the file

The check that decides is the running system, not the artifact on disk:

systemctl show -p ExecMainStartTimestamp <unit>   # when the process actually started
stat -c '%y %n' /path/to/the/config/file          # when the file actually changed

If the process started before the file changed, the process is running the old configuration — whatever the file says. Where a service can report its loaded value, read that instead of the file: a runtime probe that echoes the loaded configuration is a claim about the process, and the file is only a claim about the disk.

The principle

“The file changed” and “the running system changed” are different claims. Only the second is what you meant when you opened the editor, and only the second is worth putting in a change record: a file in the store is an input to a deploy, and the process that started after it is the evidence that the deploy did something.

What you can check

Pick a service, change something in its configuration, deploy, and see whether ExecMainStartTimestamp moves. If it does not, your change is on disk and not in the process.
restartTriggers for hermes-agent in hosts/common/users/hermes/default.nix (covering vpn and public) and inline in hosts/giga/default.nix, in the nix-config repository.
systemd.services.unbound.reloadIfChanged = true in hosts/mail/default.nix and hosts/giga/default.nix — reload instead of restart, to keep the resolver cache.
The Caddy note about new tls file pairs in hosts/mail/default.nix, which explains why “reload” is not always enough.

For how a deploy is verified on the host after a green run, see how the stack is built.

VictoriaLogs does not de-duplicate

2026-10-07T00:00:00-04:00

VictoriaLogs is a log store. That sounds like a description, but it is a design constraint with consequences, and the fleet’s own operations notes record the exact one: VictoriaLogs does not de-duplicate (the upstream item is VictoriaMetrics#5080, cited as open). So anything that hands it the same line twice gets two lines back.

That is not a bug report. It is the behaviour a store should have. The point is that two ordinary design facts become duplicated history if you do not account for them.

A retrying agent re-sends a batch

The file collector here is vlagent, configured in modules/monitoring/vlagent.nix. It writes to a disk buffer and keeps a checkpoint:

-remoteWrite.tmpDataPath=/var/lib/vlagent
-fileCollector.checkpointsPath=/var/lib/vlagent/file-checkpoints.json

That is a correct, durable design — the unit even restarts on failure with Restart = "on-failure" and RestartSec = "5s". But a batch that was written to VictoriaLogs and then not acknowledged — the connection dropped, the process restarted mid-flight — is re-sent from the checkpoint. Because the store does not de-duplicate, that is not an idempotent replay. It is a second copy of the same lines.

A restart loop multiplies every startup line

Every unit that logs on start logs again on every restart. A service that crash-loops therefore multiplies its own startup output, and the fleet’s own notes record the shape of it: a preStart that touches a spool file without the directory existing loops on “Permission denied”, and the unfixed loop floods VictoriaLogs with thousands of “Failed to start” entries, dominating error-level log counts. The store is faithfully recording a system that is genuinely failing to start thousands of times. The store is not the thing that is wrong.

The shutdown window

The third fact is subtler: a service can log so heavily while stopping that it dominates a time window, and a stop that is held open by a stalled log upload keeps the process alive to log more. This fleet hit exactly that with systemd-journal-upload, whose POST had no timeout and could block the unit for the whole 90 s stop cap. Bounding the transfer changed the shape of the shutdown: the notes record the measured improvement on the giga host — 90 s → 11 s (stop cap only) → 6 s once NetworkTimeoutSec was set, with the shutdown window’s tail complete in the store (847 on-disk lines, at least 2000 in VictoriaLogs).

The same note records the cost of not bounding it: because the store does not de-duplicate, a SIGKILLed stop can re-ingest up to one state-save interval, measured at roughly one duplicate line. Small — and exactly the kind of number that stays invisible until it is not.

Rules

“Is this line cheap to store?” belongs in the logging decision. A line written once per loop iteration is not the same as a line written once per state change.
Prefer one line per state change over one per iteration. Log the transition, not the tick.
Remember the replay is not free. A disk-buffered retry is durable; it is not idempotent.
When something looks slow, check ingest rate and per-stream volume. A restart loop and a replayed batch both show up as volume, not as an error message.
Ask what you are actually shipping. The question is not whether the store is correct — it is whether the volume arriving is the volume you intended to ship.

What you can check

modules/monitoring/vlagent.nix in the nix-config repository for the tmpDataPath buffer, the checkpoint path and the Restart policy.
VictoriaMetrics#5080, the de-duplication item the fleet’s notes cite as open.
The measured shutdown figures and the one-duplicate-line cost, in the victorialogs-vmagent operations notes (the systemd-journal-upload section), alongside the systemd-journal-upload.service: State 'stop-sigterm' timed out. Killing. line that starts the diagnosis.

This is the same argument as the logging posture page: what you ship decides what is in the store, and the store keeps what it is given.

Proportional weights are not caps

2026-10-06T00:00:00-04:00

A CPUWeight looks like a way to promise a service some CPU. It reads like an allocation: give the important thing a big number, give the backup a small one, and the important thing is protected. That reading is wrong, and the module header in this fleet’s config says so in its own words — the weights “are not caps — on an idle box the background unit still gets everything.”

The file is modules/service-priority.nix. Its header is worth reading before the code.

A weight only decides who wins a tie

cgroup v2 CPUWeight is a relative weight, defaulting to 100, and it is consulted only when two cgroups want the same resource at the same moment. One unit at weight 1000 beside a unit at weight 10 means the first gets a hundred times the second’s share of whatever is contended. It does not mean the first gets a hundred CPUs, or even one. On an idle machine the low-weight unit runs as fast as it likes, because nothing is competing with it. IOWeight is the same idea one layer down, for the block layer.

This fleet sets CPUWeight = 1000 and IOWeight = 1000 on the client-facing units, and 10 and 10 on the build and backup units, and the header is explicit about why that is not a CPUQuota: on an idle box “the background unit still gets everything, so backups/builds do not slow down for nothing.” That is a deliberate trade — fairness, not protection.

What the numbers were, measured

The header records a real measurement from 2026-09-19 on the public host, a 1-vCPU edge: a fleet restic run pushed a portal fetch from 0.25 s time-to-first-byte to 1.60 s, and from 1.41 MB/s to 221 KB/s on the same URL, while a CI build held CPU pressure at roughly 58%. A 6x latency regression with no configuration error anywhere. A neighbour got busy, and the proportional share did exactly what it says on the tin.

The method matters more than the numbers: drive load on one unit, then measure the latency of one known request on the other, on the same host and the same URL. A weight setting is not verified by reading it. It is verified by measuring both sides of the contention.

The load you contained may not be in the cgroup you contained

The second trap is quieter, and it is the one that survives a correct weight table. Builds run in the nix-daemon’s cgroup, not the runner’s: the runner only spawns the nix client, and the daemon forks the builders. So putting a CI runner in its own slice and weighting that slice does not bound the builds it triggers — they are the daemon’s children, weighed wherever the daemon is. The module weights nix-daemon.serviceConfig too, and at 50 rather than 10, because the deploy jobs also go through that daemon and a deploy must not crawl.

That is the general lesson: before you claim a workload is isolated, ask which cgroup it actually lands in.

systemd-cgls                 # the tree, with units under their cgroups
systemctl status <unit>      # the unit's own slice and cgroup

Rules

Weights are for fairness. Use them to decide who wins when siblings compete, and expect the loser to take its full share when nothing else wants the resource.
Quotas and limits are for protection. CPUQuota is a ceiling. MemoryMax is a hard cap. If a latency-sensitive service must not be slowed by a greedy neighbour, that is the tool.
Verify the cgroup, not the intent. Confirm with systemd-cgls or systemctl status that the work lands where the weight is before you claim isolation.
Know what cgroup weighting cannot reach. The same header names two limits: IOSchedulingClass is inert on these hosts — the only block device is a rotational volume on mq-deadline, which does not honour io priorities — and on the wireguard host the client dataplane is kernel-side (wg-crypt kworkers, NAT forwarding, softirq) and cannot be cgroup-weighted at all.

What you can check

modules/service-priority.nix in the nix-config repository: the “not caps” paragraph, the CPUWeight = 1000 / 10 split, and the nix-daemon weight of 50.
The MemoryHigh = "768M" on the background units and MemoryLow = "128M" on the foreground ones — on these hosts the actual disk-latency protection comes from the memory side, not from the weights.
On a host: systemd-cgls to see which cgroup a build actually lands in.

For how a deploy is verified on the host rather than assumed from a green run, see how the stack is built.

A speed test that runs inside the machine it measures

2026-10-05T00:00:00-04:00

A throughput number is a claim about a path, and a path has two ends. If both ends are the same host, or if one end is a one-vCPU machine that is busy running everything else, the number is mostly a measurement of the machine. It will happily report a fast link over a bad one. The header of internal/api/speedtest.go puts it bluntly:

// a speed test running inside the machine it is measuring is not a speed test

Where the bytes actually travel

The browser in the tunnel moves bytes to and from the tunnel endpoint — the host every client tunnel terminates on, served by this same binary under its clientsv probe role. The downlink is streamed from there; the uplink is counted there.

The portal is deliberately the other half. It serves the page, owns the session, mints the short-lived proof the far end requires, and writes the one result row. It does not move the bytes. That distinction is the design: the edge host is one vCPU that also serves the site, the admin console, monitoring and CI, and its load moves the throughput measured through it by a factor of several.

The far end is a deliberately minimal role of the same binary: no database, no session, no cookies, no templates, no history, no write path. Two routes carry the load, and both are pure copy work:

block := speedBlock()                          // 1 MiB, incompressible, repeated

n, err := io.Copy(io.Discard, io.LimitReader(r.Body, cap+1))   // the uplink

Every byte the test moves is a byte the host copied: a one-vCPU host can saturate on the copy before the link does, so the ceiling is often the host, not the network. A finished test stores one row — two rates, the byte counts, the latency, the device and tier, and the time.

Auth without a usage log

An endpoint anyone can hit has to be authenticated, and the easy way is to give it an account and log it. A session cookie is host-only, so a browser cannot carry the portal’s session to another name — the far end needs its own credential, and it gets a short-lived bearer token instead of an account.

The portal mints it and the far end verifies it statelessly: HMAC-SHA256 over an expiry and a nonce, keyed by a secret both hosts read from the same sops file. The nonce makes two tokens minted in the same second different, so a leaked one is not replayable. The shipped TTL is 3 minutes, the far end accepts nothing with an expiry more than 15 minutes out, and 30 seconds of clock skew is absorbed.

There is no access log entry for any of it. The far end writes nothing, the fleet’s caddy skips the probe paths on the endpoint, and on the edge /api/speedtest/* is log_skipped next to a comment explaining why. One request line carries a byte count and a timestamp, and a few dozen of them are a per-test throughput measurement tied to an address.

The induced load is itself a denial-of-service surface on a small edge host, which is why the credential is short-lived and per-request rather than a long-lived key, and why there is a concurrency ceiling of 12 requests in flight. Over the ceiling the far end answers 503 and the page falls back to measuring at the portal — a stalling client degrades the test instead of parking the host. Stealing a token buys bandwidth, not access, and it buys it for minutes.

What a throughput figure needs next to it

Never present one number on its own. Say, at minimum:

What the two endpoints were. “Client to tunnel endpoint” and “client to the site’s front door” are different paths and different numbers.
What the transfer size was. The test streams a 1 MiB block repeatedly and caps a single request at 512 MiB per direction; a short transfer measures ramp, not throughput.
What the host was doing. On a one-vCPU edge, a backup job or a deploy moves the number by a factor of several.

What you can check

The headers, in internal/api/speedtest.go and internal/api/probe.go: where the bytes travel, what is stored, and the sentence about the machine measuring itself.
The token. MintProbeToken and probeTokenValid show the HMAC over expiry and nonce, the 3-minute TTL and the 15-minute ceiling; the key comes from the same sops file both hosts read.
The absence of a log line. /api/speedtest/* is log_skipped in the public vhost’s caddy config, with the reasoning written next to it, and the probe paths are skipped on the endpoint.
The one row. Its fields and its five-row retention are in the same file as the code that writes them.

A short-lived token protects against replay, against correlation by value, and bounds the blast radius of a leak — it expires in minutes and grants a bounded transfer to one far end. It does not protect the host from a concurrent test saturating its CPU, which the concurrency ceiling caps as a degradation control rather than a defence.

What is recorded and what is not is on what we log; the policy it belongs to is what “no logs” can and cannot promise.

Shape it, don't meter it: fairness with no per-peer record

2026-10-04T00:00:00-04:00

A saturated link raises two questions that look like one: unequal service, and who caused it. The second is a logging question, and here the answer is no. There is no per-peer traffic record, so a saturated tier cannot be diagnosed after the fact and per-byte billing is off the table.

That is not the end of fairness, because fairness comes out of the data path. The file that says so starts with a comment:

// WHY A SHAPER AND NOT METERING: the no-log posture forbids keeping a per-peer
// traffic record ... Fairness therefore has to come out of the data path itself

That is internal/tiershape/tiershape.go in the clientsv repository.

Scheduling is a decision, not a ledger

HTB builds a hierarchy of classes, and a parent’s rate is shared among its children by weight. The property that matters is that HTB is work-conserving: an idle member is never dequeued to, so its share goes to the busy ones per packet. Three members saturating and one idle, and the busy ones get the idle member’s share — because the scheduler picked the next packet off whichever queue had one.

There is no counter to persist, which is exactly why the design survives a no-log posture. And you cannot report per-peer bytes from it. Asked to graph who used what, the answer is that this design does not know, and cannot be made to know without adding the record it exists to avoid.

One tree on one device

The cap a customer buys is per account, and an account may hold several devices on several tunnels, so a tree per tunnel cannot express it. The tree therefore lives on a single shaper device, an IFB, and both tunnels are funnelled into it.

Linux can only shape traffic it can enqueue, which is why an egress qdisc polices egress and a tunnel’s ingress — the client-to-server, upload direction — used to be unshaped. Redirecting both directions into a virtual device puts them under one trunk, one account class and one device class:

Rate       string   // guaranteed rate of one account
Ceil       string   // the cap one account's devices share
RootCeil   string   // trunk ceiling (the instance's capacity)

Classes are keyed on the peer’s assigned tunnel address — /32, matched as src_ip for upload and dst_ip for download — because when traffic leaves a tunnel the peer’s public key is long gone and the tunnel address is the only stable identity left. Classid tokens are base 16 with a stride of ten, so 120 is account 12’s class and 123 its third device.

What a rate is not

A guaranteed rate is a long-run average enforced by the scheduler, not a hard per-instant cap. An account may burst well above its guaranteed rate while the trunk is idle, and it can always use its guarantee even when the trunk is busy. The code pins the quantum to 1500 bytes for exactly this reason: left at tc’s default, a 1.8 Gbit trunk gets a multi-megabyte quantum — larger than the queue itself — and the scheduler then interleaves accounts per burst instead of per packet. Hitting the ceiling does not produce a row; it produces a queue that drains slower.

Checking the discipline on a live host

The answer has to come from the kernel, not the config repository. Read it in this order:

tc qdisc show — all devices, no dev filter. A real tree shows up here even when the interface you asked about looks bare.
Per interface: tc qdisc show dev $I, tc class show dev $I, tc filter show dev $I.
ip -br link show type ifb, then dump the tree on that device too — the mirror target is where a cross-interface policy actually lives.

WireGuard and AmneziaWG devices default to noqueue, so an empty tc class output on a tunnel is not proof of “unshaped”: a cross-interface policy shows there as a clsact qdisc plus filters that mirred ... redirect into the IFB. Grep the filter output for mirred before concluding. And tc -j reports HTB rates in bytes per second — rate 100mbit reads back as "rate":12500000 — so comparing against a bit value makes every class look drifted forever.

The reconciler reads the kernel back after applying: -dry-run prints the commands and changes nothing, -check reports drift as exit code 3 so a timer can alarm without mutating the tree.

What you can check

The tree, on the host. The commands above, run on the endpoint that carries the tunnels.
The header comment in internal/tiershape/tiershape.go, which states the no-log constraint and the account-versus-tier reasoning the tree follows.
The defaults — 100mbit, 1gbit, 1.8gbit, quantum 1500 — as constants in the same package.
What a no-log design cannot measure: per-peer usage, in any direction, at any resolution. The aggregate counters disclosed at what we log are the only volume data that exists, and they are attributable to an account, not to a peer’s activity.

For the policy side — what the service records and what it refuses to — see what “no logs” can and cannot promise, and for the rest of the network posture, three DNS postures, one tunnel.

The secret that reached the transcript

2026-10-03T00:00:00-04:00

The interesting question about a secret store is not whether it is encrypted. It is where the plaintext has to exist once the system is running, because that is where it gets out.

This fleet’s secrets live as ciphertext in version control — as they did in a public repository until 2026-09-23, when the single store was split by recipient set and moved to a private one. That part is sound either way, and it is worth understanding exactly why — the failure mode that actually happened here is not in the repository at all.

Ciphertext is the easy part

.sops.yaml lists recipients by age public key and maps a path pattern to the set allowed to decrypt. Until 2026-09-23 there was one store, in the public nix-config repository, and one catch-all rule over it:

# until 2026-09-23, in the public repository
creation_rules:
  - path_regex: secrets/[^/]+\.(yaml|json|env|ini)$
    key_groups:
    - age:
      - *admin_truva
      - *hermes_agent
      - *giga
      - *mail
      - *public
      - *vpn

That single rule is the defect the split fixed: one file, six recipient identities, so mail’s host key opened the portal’s tokens and the VPN’s private keys. The store is now one file per set, and a file’s name is its recipient set — secrets/shared-mail-vpn.yaml is readable by mail, vpn, the owner and the operator, and by nobody else — in the private configuration repository. What the public .sops.yaml still carries is each host’s public identity anchor, which is the complete answer to “who can read a given secret”, plus one deliberately bogus secrets/demo.yaml kept as a worked example.

Every value in the store is ENC[AES256_GCM,...]. The ciphertext is in git, nothing is exposed, and the file’s name states who can open it. Each host generated its own age key on the host, wrote the private half to /var/lib/sops-nix/keys.txt (root:root 0600), and published only the public half. The headers record two rotations: a full one on 2026-09-17, and the 2026-09-23 re-key whose identities superseded those. The second matters because the escrow ciphertext of the 2026-09-17 identities sits in the public repository’s history to this day — which is why the fleet was re-keyed rather than merely re-encrypted. No retired identity can recover a current value from any revision.

That is the positive control, and it is the part people miss: because the recipients are enumerated and the store is in git, you can say precisely who could read a given secret, and rotating is a reviewable commit rather than an incident channel.

But the plaintext has to be rendered somewhere

sops-nix decrypts at activation, not at build. The plaintext then lands in the running system in the shapes a unit needs: an EnvironmentFile, or a templated file under a runtime path. Both are ordinary configuration:

environmentFiles = [ config.sops.secrets.hermes-env.path ];   # hosts/giga/default.nix

So the plaintext is reachable by anything that can read that rendered path, or that dumps a unit’s resolved environment. A diagnostic that prints a unit’s resolved configuration prints secrets. A transcript that captures such a command pushes those values off-host, and a transcript has no expiry.

The scar this policy has

Two records are kept in the repository itself, which is where they belong.

The first is in .sops.yaml: the pre-rotation identity list is described as the four ssh-host-key-derived keys, and the header says, in a parenthetical, that mail’s had leaked into a transcript. A host key used as an age identity is a host key that can be extracted from wherever it was printed.

The second is an assertion. sops.age.generateKey = true makes sops-install-secrets generate an age key at activation and print its secret half to stdout — and the deploy jobs run nixos-rebuild switch with stdout and stderr going straight into the job log, which any reader of the repository can open. The guard in hosts/common/global/sops-generate-key-guard.nix fails evaluation rather than publish a key in production: assertion = cfg.allow || !(config.sops.age.generateKey or false);

The scale of the transcript risk is not hypothetical either. A bare sops -d on a store file returns every value. When that happened here the single store held 48 live secrets: one tool result, off-host to the model provider, with nothing left but a full rotation. The operational rule is the narrow one: use sops -d secrets/<set>.yaml | cut -d: -f1 when you want key names.

The net under the rule

modules/hermes-plugin-secret-guard exists because Hermes’ own redaction catches credential-shaped strings, and a sops value is opaque — a passphrase, a PreSharedKey, a plain hex token — so it passes through untouched. The plugin has two hooks: transform_tool_result rewrites every known value in any tool result to [REDACTED:<name>] before it reaches the model, the session store or the logs, and pre_tool_call notices commands that print raw values (sops -d, cat /run/secrets/..., age -d, printenv ...TOKEN). Its own README states that the incident it prevents has happened twice.

A guard is a net, not a policy. The policy is four rules:

Read the encrypted source and the metadata — key names, recipients, rotation date — rather than the rendered value.
Identify a secret by name and version, never by value.
Treat transcripts, issue bodies and CI logs as publication surfaces, because they are.
Rotate anything that was exposed. “It was only a transcript” is not a threat model.

What you can check

cat .sops.yaml in the public nix-config repository: every host’s public identity anchor, and the header that dates the split and the rotations. The store’s own rules are in the private repository; there, a file’s name is its recipient set.
The rendered paths on a host. systemctl show hermes-agent -p Environment, and the file behind any config.sops.secrets.*.path — this is what a diagnostic prints, and the reason the quoting rule exists.
The assertion. hosts/common/global/sops-generate-key-guard.nix fails a build if a host is configured to generate its own key at activation, with the log-reading reasoning in the failure message.
The key names only. sops -d secrets/<set>.yaml | cut -d: -f1 returns identifiers, which is the whole of what a review needs.

This is the same argument as what “no logs” can and cannot promise: a policy is meaningful when you can name the surface it protects. For how the store is wired into the hosts, see how the stack is built, and for the automated check over the repository, the audit page.

The auto-merge that dropped a security gate

2026-10-02T00:00:00-04:00

Auto-merge does not automate the judgement “this change is correct”. It automates a different sentence: “this change was correct at some earlier moment”. Those are the same statement only while the base branch holds still — and on a repo taking several pull requests an hour, it does not.

What actually landed

The failure on this fleet was not a bad merge in any dramatic sense. It was textual. Git can auto-merge a workflow file “successfully” and still drop a job’s steps, because a merge is textual and the two edits were structurally different. Measured here: a rebase over a concurrent pull request silently removed a signature-verification gate from the one job that actually publishes — a green pipeline with a missing security gate.

Nothing about that outcome is visible in the result. The file has no conflict markers, the pull request is green, and the job that lost its gate still runs — without the thing that was protecting it.

A conflict resolution is worse, not better: it is new code that no test has seen. The merge commit is not the commit that passed CI, and most CI systems do not build the merge result unless you explicitly ask them to. So the artifact that reaches main is frequently a revision nobody evaluated.

Required checks are evaluated against a base that can move

The second half of the problem is in the forge. Required status checks are evaluated against the base branch as it was when they ran — and Forgejo does not re-run pull-request workflows when the base branch moves: it runs them on pull-request events (opened, synchronized, reopened). Nothing else.

A pull request that was green against main an hour ago therefore still shows green after several other pull requests landed underneath it. The check is a true statement about an earlier main. The workflow that exists because of this is explicit about it:

Forgejo does NOT re-run PR workflows when the BASE branch moves: it only runs them on PR events (opened / synchronize / reopened).

That comment sits at the top of .forgejo/workflows/pr-base-guard.yml in the public ymrtech/nix-config repository. The job inside re-checks the pull request head against the current origin/main using the same three-way merge the merge itself would perform — git merge-tree --write-tree — so a real conflict fails in seconds instead of surfacing at merge time as a 405, or after the merge as a post-merge deploy rollback.

Three details are worth stealing:

A required context is the workflow name, a literal ` / `, the job name, then the event in parens. The suffix is part of the string, so renaming a workflow or a job silently breaks enforcement — the required context then never reports.
Only unfiltered jobs may be required. A paths:-filtered or if:-skipped job never posts a passing status, so requiring one blocks every merge forever. Exactly two jobs on this repo qualify and are required: Secret Scan / gitleaks scan (pull_request) and PR base guard / Base drift / conflict check (pull_request).
It deliberately does not fail a behind-but-clean branch. With several pull requests landing per hour, requiring branch-equals-main would invalidate every open pull request on each merge. It covers the real risk — a conflicting branch — and warns when main moved and touched the same files.

Make the safe path the cheap one

The counter-argument is that auto-merge exists because the manual step is boring, and a boring step that is merely required gets skipped. Discipline is not a control. So the fix is not “be more careful” — it is to leave auto-merge in place and put the verification where the work already happens.

The invariant is short: the gate must be re-evaluated against the merge RESULT, and a human reads every conflict resolution. The base guard is that idea made cheap — one [gate]-labelled job, no nix, seconds of runtime, unfiltered, so it runs on every pull request.

The other half is refusing to trust the merge call — this chain learned it the same way:

FATAL: PR #$LOCK_PR reported merged but main's flake.lock does not pin clientsv@$MYSHA

The comment above it records why: a 200 from the merge endpoint without the merge applying was reported as success, because the wait that followed matched a pre-existing deploy of the old main head. The lock never reached main and nothing noticed. The remedy has the same shape as everywhere else on this site — assert the effect, not the status code.

What you can check

Whether your forge re-runs checks when the base branch updates. Most do not, and the consequence is that “green” is a claim about a main that may have moved. Test it rather than assuming: push a second pull request that changes a file the first one touched, and see whether the first one’s checks re-run.
Whether a recent merge commit differs from the commit CI actually tested. If the revision that landed on main is not the head that went green, no check saw what you shipped.
Whether a conflict was resolved by a person. A resolution is code; it needs the same reading as any other.

This is the same argument as a green deploy that deployed nothing: the check is about a particular revision of a particular tree, and that deploy had computed its scope from a single commit’s diff. For the gate that verifies a change after it lands — the health check and the automatic rollback on the host itself — see how the stack is built.

When the tool shows you ***, never patch from the read

2026-10-01T00:00:00-04:00

A display is not a file. That sounds obvious until you notice most automation never sees the file — it sees a rendering of it, and writes that back.

Two things make a rendering lie without ever being broken. One is masking, applied on display. The other is truncation, applied to every tool that pipes command output into a summary. Both leave the underlying file intact. Both produce an edit that is wrong the moment you trust them.

Masking replaces the value on the way out, not in the file

The fleet runs a secret guard whose whole job is to keep live values out of a transcript. Its central hook, transform_tool_result, replaces every known value in any tool result with a [REDACTED:…] marker naming the key, before that result reaches the model, the session store or the logs.

The file on disk still contains the real value. The transcript contains a placeholder. Build an edit from the transcript — “replace the value I saw with the new one” — and the value you saw is the placeholder, so the placeholder is what gets written. The secret is now a broken literal in a secrets store, and nothing failed.

The same shape shows up in CI logs. This fleet’s runner honours ::add-mask:: — probe run 1647 printed the literal as *** — and the deploy workflow pipes every switch, rollback and journal command through a mask() filter that rewrites AGE-SECRET-KEY-… to [redacted-age-key]. So *** is not a hypothetical; it is a rendering this infrastructure produces, in two different places, on purpose.

Why the guard exists sets the stakes. An agent once ran sops -d secrets/secrets.yaml “just to list the key names”, the command returned every value, and all 48 live fleet secrets were in the conversation — already sent off-host. Nothing undoes that, and the only remedy left was rotating everything. Built-in redaction does not catch it either: it matches credential-shaped strings (sk-…, AKIA…, high-entropy blobs), while sops values are opaque passphrases, pre-shared keys, PEM bodies and plain hex.

Making the transcript safe is the guard’s job. It is not a licence to edit from the transcript, because the transcript is the thing that was falsified.

Truncation has the same failure with no mask involved

Command output is auto-truncated with the full text saved to a file, and large reads return a continuation offset instead of the rest. The tool tells you — there is a footer saying so. That footer is also the easiest thing in the output to scroll past.

An edit keyed on a truncated read has the same defect as one keyed on a masked value, one step further on: a replace-all against the portion you could see silently drops everything past the cutoff. Nothing errors. You replaced every occurrence you were shown.

The procedure that avoids both

Re-read the exact region immediately before editing. Not the file you read five minutes ago and not the summary of it — the current lines, fresh.
Key the edit on unique surrounding context, never on the value. Anchor on the neighbouring key name, the line above, the structure around it. The value is the one part of a secret’s entry you were never meant to hold; the key name is public.
Change a secret by name reference inside the encrypted store, not by replacing a literal. This is where sops helps, and where the .sops.yaml creation rules decide the outcome by path. Since the store was split one file per recipient set there is no catch-all rule: each store file is matched by its own path_regex: secrets/<set>\.yaml$, alongside one rule for the secrets/age-escrow/*.age identities. Encrypt a temp file outside secrets/ and no rule matches: sops --encrypt produces a plaintext file with no SOPS metadata, throws no error, and exits 0.
Verify the encryption, not the command. After any edit the key must read back as ciphertext — ENC[ — not as a plaintext value. The reliable shape is decrypt into secrets/, edit, sops --encrypt --in-place on a file still inside secrets/, verify, then replace.
Inventory by name only. sops -d secrets/<set>.yaml | grep -oE '^[a-zA-Z0-9_.-]+:' prints the matched part, which is the key name — values never reach output. To answer “did this change / is it non-empty”, compare digests and counts (sha256sum, grep -c, wc -c), never values.

The review habit that catches it after the fact

Any diff containing ***, ..., or a run of dots in a value position was built from a rendering, and should be rejected on sight. So should a diff where a secrets file’s value position has lost its ENC[ envelope. Both are cheap to check now and impossible to check later.

What you can check

Whether your tooling masks on read. If a value arrives as *** or a bracketed name, that tool is telling you the file was not what you received.
Whether the file on disk still holds the real value. Read the name, not the value: a count or a digest comparison.
Whether a recent diff touched a secrets file and contains a placeholder. That is a one-line search.

The layer underneath — where the encrypted store lives, who the recipients are, and how a host decrypts at activation — is on how the stack is built. The same discipline runs through the audit work: a claim you cannot inspect is a claim about intentions, which is why the audit page publishes signed artifacts rather than a status.

The token that cannot see the other repository

2026-09-30T00:00:00-04:00

A cross-repository step that does nothing is rarely loud. It usually fails as an authentication symptom wearing a not-found costume, and on this fleet there is a specific token that cannot see the other repository — by design, because it is scoped to one repo.

The step that has to reach across

ymrtech/nix-config-private deploys by bumping its own input inside the public ymrtech/nix-config repository’s flake.lock. Pushing to the private repo alone changes nothing on any host until that lock moves. So the private repo’s CD chain has to write a branch and open a pull request in a repository it does not own — and the workflow token cannot do it. The workflow says so in its own env block:

env:
  NIXCFG: ymrtech/nix-config
  API: https://git.ymrtech.com/api/v1
  # github.token is repo-scoped (cannot write nix-config); use the runner
  # user's ~/.netrc (hermes PAT, rendered on every runner host by the public
  # repo's modules/clientsv/nix-netrc.nix).

Two facts follow, and both are load-bearing.

A repo-scoped token answers 404 for a repository outside its scope. Not 403. The API returns the same status it would for a repository that does not exist, so “you are not allowed” and “there is nothing there” are indistinguishable in the log. This is not one endpoint’s quirk: on the same forge, the same netrc token produces nix-config -> 200 ("private":false) and nix-config-private -> 404 (denied). Private-repo run pages and job logs behave the same way — token auth covers the public repo’s pages only.

netrc matches by host. The credential file on this runner has exactly one entry: machine git.ymrtech.com. A lookup with curl -n only finds credentials when the URL’s hostname matches, which is why the CI comment is explicit about it — curl’s netrc lookup matches the HOSTNAME, so you always call the public hostname the netrc knows, never an internal address for the same service.

Why it can exit zero and change nothing

Together those two give you a script with a working read path and a dead write path — the worst combination to debug, because the first half looks like proof that the credentials are fine.

Public repositories need no read auth at all. Resolving main’s head, listing open pull requests: all of it succeeds with no token, or with a token that has no permission over that repo. Every step before the write looks healthy.
Then the write 404s. The step that creates the branch or the pull request fails at the one operation that needs scope — and the message is a not-found.
curl exits 0 on an HTTP error unless you ask it not to. Without -f, curl hands the error document to jq, and a pipeline that merely checks the command ran can conclude that it did. A 404 page not found body is valid input to a jq expression that then yields null, and null reads as “no result”, not “no permission”.

So the step runs, prints something plausible, exits zero, and changes nothing. The only symptom is the absence of an effect.

The fix is to assert the effect

Not to widen the token. A token that can write to everything removes the symptom and keeps the bug — the step was never verified, it just stopped failing. Scope the token to what the chain needs, then check the result directly.

This chain does, because it learned the hard way. Its merge step re-reads the artifact after merging and compares it against the exact revision this run pushed:

FATAL: PR #$LOCK_PR reported merged but main's flake.lock does not pin clientsv@$MYSHA

The comment above it records the lesson: a 200 from the merge endpoint without the merge applying was reported as success, because the wait that followed matched a pre-existing deploy of the old main head — the lock never reached main and nothing noticed. A status code from the command that claims to have done the work is not the work. The pull-request check follows the same rule: rather than trusting the POST that created it, it lists open pull requests and matches .head.ref against the per-revision branch name it just pushed.

The general form: assert that the pull request exists at the expected head revision, or that the file at ref=main contains the value you set, or that the branch resolves. Something that can only be true if the effect happened.

What you can check

Your netrc’s machine lines. Every hostname you call must appear, spelled the way you call it. A missing entry is not an error; it is silence.
Whether your log can tell the two 404s apart. It cannot, from the status alone. Read the body, and treat any cross-repo step as unverified until you have checked the artifact.
Whether the step asserts its effect, or merely reports its exit code. Find the write step and ask what verifies it afterwards.

This is the same class of problem as a green deploy that deployed nothing: a status is an input to a decision, not the decision. That post is here; the layer underneath — the hosts, the runner, and the health gate that verifies a deploy after the fact — is on how the stack is built.

A job that waits must not hold the slot it is waiting for

2026-09-29T00:00:00-04:00

A failed job announces itself: it goes red, names a step, prints a reason. A hung job does none of that. It sits in running holding a runner slot, waiting jobs pile up behind it, and nothing in the pipeline says anything is wrong — from the pipeline’s point of view, work is in progress. What follows is the structural rule that produces it, and the parts of this fleet’s deploy graph that embody it.

One daemon, one job

The capacity of this fleet is not a policy setting; it is a property of the runner software. One forgejo-runner daemon serves one job at a time — that is the tool’s own default, and nothing in the repo raises it. Grepping the fleet configuration tells the story plainly:

grep -rn 'capacity\|instances\|runner' nix-config/modules nix-config/hosts | head -40

There is no capacity setting anywhere in nix-config/modules or nix-config/hosts; the only hits are unrelated (an inference-server flag and a Grafana alert comment). What the configuration sets is services.gitea-actions-runner.instances.ymrtech — one instance per host — plus the labels it advertises. The labels differ per host; the count never does. One daemon, one slot.

The workflow comment says it in as many words. deploy-rev’s own header in .forgejo/workflows/nix-deploy.yml explains why that job moved off ci: on the shared ci pool it sat “behind mail’s single capacity-1 slot”, and after a lock-bump merge it measured 3 min 10 s of pure queueing for about two seconds of git work. A label is a matcher, not capacity.

The graph, and the one edge that is a real dependency

.forgejo/workflows/nix-deploy.yml fans out into three per-host deploy jobs:

deploy-mail — runs-on: [deploy-mail], needs: [scope, deploy-rev], gated if: contains(needs.scope.outputs.hosts, 'mail').
deploy-vpn — runs-on: [deploy-vpn], needs: [scope, deploy-rev], gated on 'vpn'.
deploy-public — runs-on: [deploy], needs: [scope, deploy-rev, deploy-vpn], gated on 'public'.

The deploy jobs are per-host and independent by design, so a single-host change does not drag the other hosts’ closures along. There is one ordering edge, and it is not a preference. Each client DNS tier splits its halves across vpn and public — primary on vpn, standby on public — so if both jobs ran concurrently one run could bounce both halves in the same second and take every client’s resolver down. deploy-public therefore waits for deploy-vpn: a real dependency earns an edge.

It also has a price. Each job builds and switches a whole NixOS system, and each demands a runner of its own, so an ordering edge widens the critical path and multiplies the slots the run needs in sequence, not in parallel: the vpn slot must free before the public slot is asked for. That is fine here because the dependency is real. Applied loosely — an edge added for tidiness, or to make logs read better — it buys a longer deploy and a larger pool for nothing.

The rule

A job must never hold the resource it is waiting for.

Concretely, for a bounded pool: if a job occupies a slot while polling for work that also needs a slot, the queue cannot drain. Nothing fails; the waiter waits, the awaited job queues behind it, and the run reports running until the job’s own timeout expires — the v15 API has no run-cancel endpoint at all, so a stuck run holds the capacity-1 runner for its whole timeout-minutes. This fleet has hit that shape and written the rule down: a chain job held a ci slot while polling for the scope job of the build check it had triggered, so it queued behind its own CI. The fix was not a longer timeout — it was to move the waiter to a lane the awaited job can still reach, and to poll the awaited job’s per-job commit status (one small call) instead of a multi-megabyte task listing computed inside the held slot.

Two rules follow from the same accounting: a dependency widens the critical path and multiplies slot demand, so it needs a reason; and because a hung run looks nothing like a failed one, queue depth is worth watching on its own. A run running far longer than normal, with other jobs waiting behind it, is the signature. Do not wait for a red mark that will never come.

What you can check

The slot count. grep -rn 'capacity' nix-config/modules nix-config/hosts returns no runner capacity setting; each host declares one services.gitea-actions-runner.instances.ymrtech. A daemon registers the sops-rendered /run/secrets/runner-config-<host>, not the Nix labels option — read the two together or you will trust a label list that is not in force.
A run’s job queue. python3 /var/lib/hermes/workspace/fjlog.py jobs <runIndex> prints its jobs with name, status and duration; fjlog.py status <sha> is the cheaper per-job liveness view. Skipped jobs leave their commit-status context stuck at pending forever, so a green run is usable while a non-terminal aggregate is not.
Slot occupancy, per host. Every host declares the daemon as systemd.services."gitea-runner-ymrtech", so its own journal on that host — journalctl -u gitea-runner-ymrtech — records what it claimed and when. The capacity lever, if needed, is act_runner’s [runner] capacity.
The habit. For every needs: edge, ask whether the two jobs could have run at once. If yes, and the edge exists anyway, it costs a slot and a serialisation for nothing. The one edge in this graph has its justification in the file: two halves of a DNS tier must not be bounced in the same second.

For the wider pool — which labels each host advertises, and why the deploy jobs stay pinned while the light gate jobs float — see how the stack is built.

One merge, two runs, the same sha

2026-09-28T00:00:00-04:00

One merge. Two runs of the same workflow. Same commit sha. Neither cancelled the other — the second one waited, then ran to completion. That is the designed behaviour of three lines at the top of .forgejo/workflows/nix-deploy.yml in the nix-config repository, and understanding it changes how you read a deploy log.

Queue, never cancel

concurrency:
  group: nix-deploy-$
  cancel-in-progress: false

Lines 151–153. The workflow’s own comment above them explains why this block exists at all. Forgejo’s documented default, for a workflow with no top-level concurrency, is that for on.push “whenever a new event triggers a new workflow invocation, any other invocations of the same workflow will be canceled automatically”. That default is right for a check — a superseded build is evaluating a commit nobody will merge — and wrong for a deploy. Cancelling a nixos-rebuild switch mid-activation can tear an activation half-way (some units new, some old) and skip the health gate and automatic rollback that follow it. The comment records a live instance: runs 2701 and 2712 cancelled about forty seconds apart while 2701 was mid-switch on vpn, leaving the fleet mixed.

Declaring any top-level concurrency block takes the workflow out of the automatic-cancellation regime. With a group and cancel-in-progress: false, a superseded run queues and completes instead of being killed.

The consequence for reading logs is direct: a later run in the same group waits rather than cancelling. Both runs execute, in order, and the last one to touch a host wins.

The group is keyed on the ref

group: nix-deploy-$ means a push to main and a workflow_dispatch on a PR branch are in different groups and never queue against each other. Two things follow: a dispatch and a push to the same ref share one queue, so the dispatch lines up behind the push rather than racing it; and runs on different refs do not serialise at all, even when they name the same host.

That second point is why the workflow also carries per-host job groups — deploy-mail, deploy-vpn, deploy-public, each with cancel-in-progress: false. As the file notes from Forgejo’s documentation, “Multiple jobs within a workflow are not affected by the concurrency setting”, so the job-level groups are what make two runs exclusive on the same host.

The job graph, and one deliberate `always()`

Job	`runs-on`	`needs`	`if`
`scope`	`[gate]`	—	—
`deploy-rev`	`[gate]`	—	—
`deploy-mail`	`[deploy-mail]`	`[scope, deploy-rev]`	`contains(needs.scope.outputs.hosts, 'mail')`
`deploy-vpn`	`[deploy-vpn]`	`[scope, deploy-rev]`	`contains(needs.scope.outputs.hosts, 'vpn')`
`deploy-public`	`[deploy]`	`[scope, deploy-rev, deploy-vpn]`	`always() && contains(needs.scope.outputs.hosts, 'public')`

Three per-host deploy jobs, one real ordering edge. deploy-public waits on deploy-vpn because the client DNS tiers split their two halves across the two hosts — primary on vpn, standby on public — and the jobs would otherwise run side by side and bounce both halves in the same second. The always() is the subtle half: without it, a skipped deploy-vpn (vpn simply not in scope) would skip deploy-public too, because a skipped job skips everything that needs: it. So the public job starts even when the vpn job failed or never ran, and its pre-flight step re-checks needs.deploy-vpn.result and refuses to switch if the primary side is broken. That is a deliberate choice about publishing the edge: the fallback halves must not be left un-deployed just because the primary tier was untouched.

Identify a run by three things, not by position

With more than one run able to touch one host, “the most recent run in the UI” is not an identifier. A run is identified by workflow + commit sha + trigger event:

The runs list items carry workflow_id, commit_sha, status and prettyref, which is enough to find the run for a specific merge per workflow — fjlog.py runs --sha <sha> --workflow nix-deploy filters on exactly those.
The trigger event is in the branch field: prettyref on a run-list item, head_branch on a task row. PR-event runs read #<PR>; push runs read the branch name. A filter that matches only one form silently misses the other and waits out its whole deadline against a run that is already finished.
A run’s status is not the state of the host. A push-to-main run can be cancelled after it already did its work — measured case: a cancelled nix-deploy whose target host had switched seven minutes earlier. Cancelled is not evidence a deploy did not land, and success is not evidence it did.

When two runs can touch one host, the generation on the host is evidence and the badge is not. nixos-rebuild list-generations prints newest first, so the current generation is the head of that list; the run’s own badge can only tell you what the pipeline intended.

What you can check

The block itself: .forgejo/workflows/nix-deploy.yml, lines 151–153 — group: nix-deploy-$, cancel-in-progress: false — plus the job-level groups at deploy-mail, deploy-vpn and deploy-public.
The gates on lines 456, 884 and 1601: the per-host contains(...) conditions, and always() && only on deploy-public.
That a superseded run queues: dispatch the same workflow twice on one branch and read the second run — it shows waiting while the first stays running to completion.
The host, not the run: its current generation and the timestamp on it.

For the scope job these deploys hang off — and why a green run can mean nothing was in scope at all — see the green deploy that deployed nothing.

Forgejo does not re-run your pull request when main moves

2026-09-27T00:00:00-04:00

A green check next to a pull request is one of the most over-read marks on a forge. It does not mean “this change is good to merge”. It means something much narrower: the commit at the head of this PR, merged with the main that existed when the run started, passed the gates. The second half of that sentence is the part that expires.

Forgejo only runs PR workflows on PR events

The check that guards every PR in the nix-config repository states the reason in its own header. .forgejo/workflows/pr-base-guard.yml, first comment block:

Forgejo does NOT re-run PR workflows when the BASE branch moves: it only runs them on PR events (opened / synchronize / reopened).

synchronize is an event you cause by pushing to the branch. A merge landing on main is an event on a different branch, so it does not re-trigger your PR’s workflows. The badge does not go stale, because the forge has no concept of a check describing an older main — from its side, the check is simply green.

What the guard computes

The guard exists for exactly that gap, and it computes the merge instead of guessing at it:

git merge-tree --write-tree --no-messages --name-only origin/main <branch>
# exit 0 = merges clean
# exit 1 = real conflict
# exit >1 = git could not compute the merge at all

The third code is the one worth reading twice. >1 means git could not perform the merge — unrelated histories being the classic case — and the guard fails that as unmergeable, with its own message, rather than folding it into “conflicts”. The remedy it prints is not a rebase: an unmergeable branch has to be recreated from origin/main and the change re-applied.

Three details make the job trustworthy:

It checks out $ — the branch head, deliberately not Forgejo’s merge ref, because a merge ref is computed against a current main and would hide the drift being measured.
It runs git fetch --quiet origin main first and checks against that, not a cached ref.
It fails closed on missing tooling: a capability probe merges HEAD with itself, and if merge-tree --write-tree is unavailable — it needs git 2.38; the fleet runs 2.55 — the job exits 1 rather than reporting a clean merge it never computed.

Conflict is the loud case; clobbering is the quiet one

A red Base drift / conflict check is the case people expect: rebase, resolve, push. The guard goes further. When the merge is clean it also compares the files your branch changed since the merge base against the files main changed in the same window, and emits a ::warning:: when they overlap. It stays green on purpose — with several PRs an hour, failing every behind-but-clean branch would force every open PR to rebase and re-run its per-host closure builds whenever anything else merges.

That warning carries the real risk. In a repository where more than one agent session edits at once, “textually clean” is not “semantically safe”. The guard’s own failure text says not to merge main into the branch as a shortcut, and to resolve on the file’s new structure rather than re-instating your copy, because in this repo a conflict usually means main rewrote that file underneath you. A merge that applies your hunk to a file someone restructured two merges ago can land green and quietly undo their change.

The merge endpoint agrees with the guard. A merge returning 405, or a PR reading mergeable:false, almost always means the branch is behind main — not that the forge is broken.

Why this is the default hazard, not an edge case

A shared, fast-moving repository turns base drift from an edge case into the normal state. When several sessions and several PRs land per hour, a branch cut twenty minutes ago is already behind, and every merge to main invalidates the evidence under every other open PR. The discipline this fleet runs on follows directly: fetch and rebase onto origin/main immediately before every push, and re-verify the base immediately before merging — the only pre-merge check that means anything is one computed against the current main.

So the pre-merge check is not the badge. It is, in the moment before you merge:

git fetch origin
git merge-tree --write-tree --no-messages --name-only origin/main <branch>

and when that returns non-zero: rebase onto the freshly fetched origin/main, re-apply the change onto the file’s new shape rather than restoring your copy, re-run the gates, then git push --force-with-lease.

What you can check

The workflow itself: .forgejo/workflows/pr-base-guard.yml in nix-config, and the same job in nix-config-private — workflow name PR base guard, job Base drift / conflict check, runs-on: [gate], on: pull_request with no paths: filter (docs-only PRs conflict too, and the build check skips them).
The job’s own output. It prints branch head:, origin/main:, merge base: and branch is N commit(s) behind origin/main before it judges anything, so the distance from main is visible rather than inferred.
The required-check context, read back from the commit status rather than guessed: PR base guard / Base drift / conflict check (pull_request).
The command, run locally, immediately before you click merge.

For the runner pools and the host graph these jobs run on — and why the guard floats on the light gate label instead of a host’s single slot — see how the stack is built.

Fail closed: when CI cannot compute the diff, evaluate everything

2026-09-26T00:00:00-04:00

The deploy pipeline decides which hosts a change is about by diffing the push or the pull request. That decision is an inference, and every inference needs a defined answer for the case where it cannot be performed. The answer cannot be “no changes”. A diff that could not be computed and a diff that found nothing are the same empty string, and only one of them means the work is unnecessary.

The inference, and its precondition

The scope job’s computation is one line:

git diff --name-only "HEAD~1...HEAD"

That command requires at least two commits of history to be present in the checkout. A fresh Forgejo checkout defaults to a shallow depth of one, so on a merge the parent commit is simply not in the repository, and the command fails with fatal: Invalid symmetric difference expression ...HEAD rather than returning an empty list. Both workflows that diff therefore pin fetch-depth: 0 on their scope job’s checkout: the deploy workflow for pushes, the build-check workflow for pull requests, where the base commit is older still.

That pin is not decoration. The failure has been observed: the eval-giga job’s checkout had no fetch-depth: 0, so on PR events git diff "$BASE...HEAD" died with fatal: Invalid symmetric difference expression ...HEAD and the job reported giga in scope: false. The asymmetry is what made it durable — push events escaped it (depth 1 still contains HEAD~1, so they fell through to the “everything changed” branch), meaning only pull requests silently skipped giga, and a skipped host looks exactly like a host with nothing to do.

Why the error does not surface

The scope step is written as an if/elif chain, and the guard that survives an uncomputable diff is spelled out in the workflow’s own comment:

Fail CLOSED. Inside an elif condition set -e never fires, and the branch below pipes the diff into grep -q .: an uncomputable diff (base object absent, shallow history) therefore fell through to HOSTS="" — a green run that built and evaluated NOTHING. Scope must widen, never silently vanish.

Both halves are the point. set -e is the reflex answer, and it does nothing here: a command in a condition is exempt, so a failing diff selects a branch instead of aborting the run. And the pipeline is deceptive in the other direction — grep -q . exists to say “is there any output”, and against nothing it says no, which is indistinguishable from “there is nothing to see”.

The fallback widens, in both directions

The evaluator’s fix is the shape to copy:

elif ! git diff --name-only "$BASE...HEAD" -- . >/dev/null 2>&1; then
  echo "WARN: cannot diff against '$BASE' - evaluating all deployable hosts"
  HOSTS="mail public vpn"

And the push path carries the equivalent guard, testing the parent commit before the diff is attempted:

no previous tip available — deploying all hosts

Unreadable history scopes to every host, not to none. The cost of the fallback is real — a wider nix build and a few minutes of runner time — and that is the correct trade against a silent no-op, because the wide case is visible in the log and the narrow case is not.

Explicit input beats inference

The strongest version of fail-closed in this repository is the one that removes the inference entirely. A manual dispatch takes no checkout and computes no diff: the hosts input is authoritative, and it is required. An empty value aborts with

hosts input is EMPTY — refusing to deploy (name hosts explicitly: mail vpn public)

and an unrecognised host name aborts too, rather than being dropped from the scope. giga fails loudly with a pointer to its own workflow, instead of quietly deploying nothing. A required input whose empty case is an error cannot be mistaken for an empty diff — which is why that branch is the pattern worth copying into any script that currently infers scope, as opposed to one that is handed scope.

Two places an empty result and a failure look identical

set -o pipefail is present in these steps, and it is necessary but not sufficient: it makes a pipeline fail when the command on the left fails, and inside a condition that only changes which branch you take. It does not abort the run. pipefail plus an elif that widens is a fix; pipefail alone is decoration.

git diff --quiet is the second trap. It uses exit code 1 for “there are differences” and a different non-zero for “I could not compute this at all”, so a test written as if git diff --quiet; then ... else ... — the form used in flake-update.yml to decide whether flake.lock actually changed — cannot tell “identical” from “cannot diff”. In that direction it errs safely (toward “changed”), but only by accident. Anywhere the same construct gates work by testing for no change, it errs silently toward doing nothing.

What you can check

Read the declared depth where it is declared. The scope job’s with: block is the authority on fetch-depth; do not infer it from the job list.
Read the scope line. scoped hosts: '...' is the whole decision. If it prints an empty list when you know you touched a host, the scope step is broken, not the repository empty. Better: assert it non-empty in the step whenever work was expected.
Confirm the fallback fired. If the log carries no previous tip available — deploying all hosts or cannot diff against '<base>', the run widened on purpose — that is the design working, not a diff that silently missed your change.

The scope line is the same one that makes a green run meaningful, which is where a green deploy that deployed nothing begins. The pipeline itself is documented in the infrastructure breakdown, and the review discipline that keeps these claims checkable is in the engineering standards.

A rollback that cannot execute is worse than no rollback

2026-09-25T00:00:00-04:00

A rollback is the code path that runs when something is already wrong. That is the whole difficulty: it executes in the exact window where you have the least appetite and the fewest options for debugging it, on a host you have just made worse. An untested rollback is therefore not a rollback with a caveat. It is a promise about a command that has never run, attached to a moment when you cannot check.

What the gate does

Each host’s deploy job switches, then runs a health gate. In order: poll the gated services every second for up to about ten seconds (a fixed sleep 40 paid the full cost on every deploy; a fixed short sleep rolls back a service that is merely still activating, because systemctl is-active reports activating as not active), check for failed units, require every declared critical service active, and compare against a pre-switch baseline of long-running enabled services. Any violation rolls back to the previous generation and ends the step with exit 1.

The failure that looks like a result

The runner’s sudo is allow-listed to exactly one command. From the workflow’s own comment on the rollback step:

The runner is allowlisted for exactly ONE privileged command - nixos-rebuild (see security.sudo.extraRules on the host). Everything else (sudo readlink, switch-to-configuration, nix-env) prompts for a password with no TTY, and a REAL gate violation then leaves the host exactly as broken as the gate found it.

A denied sudo is not a crash. It writes its own line to stderr and exits non-zero — and if the step captured its stdout in a command substitution, the substitution is what decides the story. There is no error to propagate: the assignment succeeded, and its value is empty. set -e does not fire, because nothing failed in the sense the shell tracks.

The observed output was this:

no previous generation ... cannot roll back

with empty generation variables — because the sudo readlink that produced those numbers had been denied. A permissions failure wearing a “no previous generation” costume. The host stayed on the generation the gate had just rejected, the run went red with a misleading cause, and the log said nothing about a password prompt.

The version in the repository, and its guards

The step reads the profile link without sudo — /nix/var/nix/profiles/system is world-readable, and requiring privilege to read it is what emptied the variables — and then refuses to guess:

CUR_LINK=$(readlink /nix/var/nix/profiles/system)
CURNUM=$(printf '%s' "$CUR_LINK" | sed 's|.*system-\([0-9]*\)-link|\1|')
if ! printf '%s' "$CURNUM" | grep -qE '^[0-9]+$'; then
  echo "!!! cannot read a generation number out of '$CUR_LINK' - refusing to guess a rollback target"
  CURNUM=""
fi

Three guards follow, and each one is a failure that was seen before it was fixed:

Parse, then validate. The profile link is relative (system-474-link), so a pattern expecting a leading / substitutes nothing, the number stays a non-numeric string, and a numeric filter degrades into a string compare that matches every link — at which point sort -n | tail -1 hands back the generation the gate just rejected. That happened: the rollback re-activated the broken generation, twice, and still recorded ROLLBACK_DONE=1. A deliberate gate drill on giga (run 1649) is what exposed it, and the run’s own log did not distinguish it from a success.
Refuse a target equal to what failed. If the “previous” closure resolves to the closure that just failed, the step clears the target rather than running a rollback that would change nothing.
Verify where the profile actually landed. ROLLBACK_DONE=1 is set only when the profile reads back at the exact closure that was aimed at. Otherwise the log says !!! ROLLBACK DID NOT LAND ON THE PREVIOUS GENERATION and HOST NEEDS AN OPERATOR NOW.

Non-interactive, or it is not a check

The one place the step invokes a privileged command that is not on the allowlist is the journal dump for failed units, and it is written to fail explicitly:

sudo -n journalctl -u "$u" -n 40 --no-pager 2>&1 | mask ||
  echo "(journal not readable without privilege - read it on the host)"

-n is the load-bearing flag. Without it, sudo prints its password lecture, and that lecture is what filled the section on 2026-09-16 while the actual cause stayed hidden — a wall of text that reads like an error and is not one. The pattern generalises: invoke an allow-listed (or possibly-denied) sudo with -n, check its exit status, and say plainly which of “not permitted” and “not found” you are looking at.

Loud, because quiet is the expensive option

The rollback must abort the run and name the host, not absorb the problem. Failures here are not paged — the webhook and its spool-draining crons were removed from this pipeline on 2026-09-16 and the operator reads the runs himself — so the run log is the alert channel, which is one more reason an ambiguous line costs hours.

What you can check

Read the gate step’s own lines, not the job summary: rollback verified: the profile is on <closure>, or !!! ROLLBACK DID NOT LAND, or no usable previous generation - CANNOT ROLL BACK. A rollback claim that does not name the closure it landed on is not a rollback.
Check every privileged call in the step against the host’s sudo allowlist. On mail and vpn the allowlist is the nixos-rebuild binary alone.
Re-test by dispatching a deploy, not by trusting a green run — a green run cannot distinguish a working gate from a disarmed one. The rollback only executes when something is already wrong, which is the worst place for a command that has never been executed.

The mechanics sit in the deploy workflow documented in the infrastructure breakdown, which is also where to look for how the fleet’s other gates avoid the same trap.

A secret rotation that merged green and reached no host

2026-09-24T00:00:00-04:00

A credential rotation merges to main, and the run list stays empty. Not a red run — no run. There is nothing to misread, because there is nothing there.

That is a different silent failure from the one where a run goes green and deploys nothing. The fleet deploy has two independent gates between a merge and a host, and they answer two different questions.

One filter decides whether a run exists at all

NixOS Deploy is triggered by a paths: filter (.forgejo/workflows/nix-deploy.yml in the nix-config repository):

  push:
    branches: [main]
    paths:
      - 'flake.nix'
      - 'flake.lock'
      - 'hosts/**'
      - 'modules/**'
      - 'overlays/**'
      - 'secrets/**'

That list is not scoping. It is creation. A merge touching none of those paths produces no workflow run at all — and a missing run is not a pending run. Waiting for it never resolves; there is no job that will eventually appear.

The comment directly above the block exists because of exactly this, and it is worth reading in full:

secrets/** is host-relevant, not docs-like: the sops store is SHARED code (one file, every host’s defaultSopsFile), and the scope step below maps secrets/ to ALL hosts. Without it here, a secrets-only merge is a silent no-op: no run is created, and even a forced run would scope to zero hosts.

Two halves of one sentence, and they are the whole post. No run is created is the filter. Even a forced run would scope to zero hosts is the second gate, and it fails independently — which is why a rotation bug has two chances to be invisible.

The other gate decides which hosts

Once a run exists, its first job computes which hosts the run is about, and the whole output of that computation is one line:

scoped hosts: ' public'

The shared-code branch of that computation is an explicit list:

git diff --name-only "HEAD~1...HEAD" -- flake.nix flake.lock modules/ overlays/ hosts/common/ secrets/ | grep -q .

A hit there scopes every host, with one measured exception: a flake.lock-only bump that moves just a rev/narHash scopes to vpn alone, and the guard for that path additionally requires that no input was added, removed or renamed. A path under hosts/<host>/ scopes just that host. Anything else scopes nothing, and with an empty scope each host job’s if: contains(needs.scope.outputs.hosts, 'mail') is false, every deploy job is skipped, and the run concludes green. That mechanism — a green run that deployed nothing — is the subject of its own post. It is worth keeping the three apart:

Failure	Where it happens	What you see
No run	`on: push: paths:` filter	no run; a watch that never resolves
Green run, zero hosts	scope job, empty result	`scoped hosts: ''`
Green run, wrong hosts	scope job, stale enumeration	`scoped hosts:` without the host you changed

Two lists, and they are not the same list

This is the part that makes the class of bug recur. The paths: filter carries hosts/**; the scope’s shared-code list carries hosts/common/. That difference is deliberate and correct — a change to one host’s directory should reach that host, not the fleet. Each list answers its own question, so copying one into the other is not a fix, and adding a directory to only one of them is a partial fix whose remaining half fails silently.

The rule that follows: when a new top-level directory starts being consumed by hosts, it needs an entry in both places, and the second entry is the one that is easy to forget because the first one makes CI look alive. The same class shows up one level down. The deploy workflow’s filter deliberately names config paths only — no .forgejo/workflows/** — so a workflow-only merge does not auto-deploy, and a workflow change is validated by a manual dispatch instead. The build-check workflow’s filter does include workflows, because it has to be able to validate its own changes. And a script under .forgejo/scripts/** that a workflow step calls needs that path in the calling workflow’s filter, or editing the script runs nothing.

The escape hatch, and why it is not the fix

workflow_dispatch is not subject to paths: filters at all. A manual dispatch always runs, and its hosts input is authoritative — that job takes no checkout and computes no diff, so nothing can scope it down. That makes dispatch a reliable way to deploy a host on purpose, and it is why the forced run case in the comment is still recoverable.

It is not a fix for a missing filter entry, because the failure mode is ignorance: the rotation merged, nothing went red, and nobody knows there is work to do. An escape hatch only helps the person who already suspects.

What you can check

Reconcile the lists. Read the on: block and the scope job’s shared-code list, then list the directories the hosts actually reference. Every shared one must appear in both.
Treat a missing run as a result, not a delay. If a push touched a host-relevant file and no run exists a few seconds later, the filter decided that — it is not “still starting”. Match the diff against the filter before waiting for anything.
If you must deploy now, dispatch. A manual dispatch bypasses the whole question, and names its hosts explicitly.

The same discipline as everywhere else on this site: the deploy pipeline is only as trustworthy as the claim you can independently re-derive from the files it runs from.

The green deploy that deployed nothing

2026-09-23T00:00:00-04:00

A green check next to NixOS Deploy is one of the easiest things to misread on this fleet. It is tempting to read it as the change is on the hosts now. What it actually means is narrower and worth stating exactly, because the gap between those two sentences is where the interesting failures live.

The workflow is public: .forgejo/workflows/nix-deploy.yml in the nix-config repository. It starts by deciding which hosts the run is about, and only then runs a deploy job per host.

Green means “nothing to do”, not “done”

The first job is called Compute changed host scope, and its whole output is one line in the log:

scoped hosts: ' public'

That line is computed from a single diff:

git diff --name-only HEAD~1...HEAD

— the last commit only. A path on the shared list puts every host in scope; a path under a single host’s directory puts that host in scope; anything else puts nothing in scope at all.

Changed path	Hosts in scope
`flake.nix`, `flake.lock`, `modules/`, `overlays/`, `hosts/common/`, `secrets/`	every host
`hosts/<host>/**`	`<host>` only
README, docs, anything else	none

Each host’s deploy job is then gated on its own name:

if: contains(needs.scope.outputs.hosts, 'mail')

With an empty scope, contains() is false for every host, so every deploy job is skipped. A skipped job is not a failure. The scope job succeeds, the run concludes, and the summary is green. Nothing was deployed, and nothing was wrong — the pipeline correctly decided there was nothing to do.

That is the design working. It becomes a problem in exactly one situation: when a file that a host genuinely consumes is not on either list. Then the run is green, the log says no hosts, and the host keeps running whatever generation it had before. There is no red mark anywhere, because from the workflow’s point of view there was no work.

The scope rules are a list, and lists go stale

This is the part worth internalising: the scope rule is an enumeration of path prefixes, and an enumeration is a claim about the future that someone has to keep true. The comment above the file list in the workflow says it plainly — the shared paths are the ones that “scope every host”, and secrets/** is on that list because the secrets store is shared code, not a document. It was not always on that list. While it was missing, a change to it was classified as irrelevant, produced an empty scope, and deployed nothing at all — with a green run.

So the rule of thumb is not “does CI pass”. It is:

A green deploy run tells you the scope was computed. Read the scope.

What you can actually check

The run’s own log. scoped hosts: '...' is printed before anything deploys. If you changed hosts/mail/** and that line does not contain mail, the run is telling you it did not deploy mail.
The host, not the run. The check that counts is the one on the machine itself — the generation it is running, and the timestamp on it. A run that is green and a host that is three days behind are entirely compatible facts.
The dispatch path, which does not guess. A manual dispatch takes an explicit hosts input, and that input is authoritative: the job takes no checkout and computes no diff. An empty or unrecognised value aborts the run rather than quietly scoping to nothing. When you mean “deploy this host”, say so.

This is the same argument as anywhere else on this site where a status is published: the audit page is built from signed artifacts precisely because a claim you cannot inspect is only a claim about intentions. A green badge is an input to a decision, not the decision.

For the layer underneath this one — the hosts, the runner, the health gate that verifies a deploy after the fact and rolls it back if it fails — see how the stack is built.

1,221 commits and a pipeline that did not exist: three months of the fleet

2026-09-22T00:00:00-04:00

Between 22 June and 22 September 2026 the repositories behind this site took 1,221 commits and 534 merged pull requests. At the start of that window there was no continuous integration at all: a change reached a host when someone typed nixos-rebuild switch and watched the terminal. At the end of it, thirteen workflow files — roughly 5,700 lines — stand between a commit and a live host, and every deploy either passes a health gate or rolls itself back.

This is what those three months produced on the fleet and on the VPN product, and, in the middle of it, the week the commit log changed hands.

There is one non-technical reason the pace was possible, and it deserves to be on the record: I am single and I do not have a life. The fleet has me instead. Whatever is left over goes to the commit messages.

Every non-merge commit across the four repositories, by ISO week. Bars are split by the identity that authored the commit. The amber line is 29 August 2026.

The numbers


Non-merge commits, 22 Jun – 22 Sep	1,221
Authored under the agent identity	692 (57%)
Pull requests merged	534
Workflow files at the start / end of the window	0 / 13
Hosts described by one configuration repository	4

The single largest repository, the one that declares the hosts themselves, carried 950 of those commits and 410 of those merges.

What the fleet is

Four machines — a public edge, a mail host, a management host, and a GPU workstation — are described in one Nix flake. Nothing on them is configured by hand: the packages, the systemd units, the firewall, the WireGuard and AmneziaWG peers, the DNS resolvers, the monitoring stack and the backups are all evaluated from files in hosts/ and modules/. A host is reproducible from the repository, and the deploy path is the only path a change takes to get there — there is no ssh-and-edit step in the workflow, which is exactly what made the next three months possible at the pace they happened.

June and July — making the network declarative

The early part of the window is infrastructure work that mostly does not show up in a screenshot: the mesh itself, the WireGuard/AmneziaWG module and its peer management, DNS resolution, the backup and scrub timers, and the split between what a host declares publicly and what it keeps. The interesting thing about this period is what it enabled rather than what it shipped — once the network was code, everything after it could be reviewed, reverted and rebuilt.

On 19 July the agent framework that now does most of the typing was added to the fleet as a flake input. That is the first commit in the window where artificial intelligence appears in this repository at all, and at that point it was infrastructure, not authorship: an input in flake.lock, nothing more.

August — monitoring, then a change of hands

August is where the fleet became observable. VictoriaMetrics and VictoriaLogs landed with Grafana over the top, dashboards and alert rules went in as files, and a status page began reporting from the same stores the dashboards read. Most of the month’s commit subjects are fixes to that stack: ingest auth, retention, alert routing, panels that queried the wrong labels. That is what building observability looks like in a commit log — a long tail of small corrections until the graphs settle.

Then the last week of August does something obvious in the chart above. On 29 August the agent got its own git identity, and the commits that identity signed first are not experiments: the CI itself.

The pipeline that did not exist

At the start of this window there were zero workflow files in the fleet repository. There is no gentler way to say it than that deploys were manual and the only test was someone watching.

Commit 073ce12 on 29 August changed that in a single step — feat(ci): Forgejo Actions CI/CD — per-host deploy runners, build gate, alerting — and added the build gate and the deploy workflow at once. The same day, 1b0a065 added a secret scan so that a credential cannot reach the branch in the first place. What followed is a pipeline that grew in public, one workflow at a time:

Date	What landed
29 Aug	Build gate + per-host deploy runners + gitleaks secret scan
5 Sep	Weekly flake input update, and alerting on every CI failure path
16 Sep	Pull-request base guard — block on conflicts, warn on base drift

The deploy workflow is now 2,321 lines. It is not long because it is complicated; it is long because it refuses to guess. Before switching a host it records a baseline of the long-running enabled units, then re-checks that baseline — and every critical service — once a second, up to ten times, passing the moment all of them are active rather than waiting out a fixed sleep, and rolls the host back if a unit that was running stopped, if a critical service is not active, or if a unit disappeared that the host did not declare as retired. Each host gets its own job, so the deploy of one is not the deploy of all.

Alongside it, the product repositories got their own pipelines: the client service builds, tests and deploys itself; the private repository runs a secret scan, a host-scoped evaluation and a lock-bumping deploy chain; and this site runs a build-and-test gate, a staging deploy, a promote step, and an ephemeral per-pull-request preview — the same machinery a normal software team would expect, on hardware the same team owns.

There is a supply-chain rule in the weekly update job worth naming, because it is the kind of decision that is easy to make once and hard to keep: every flake input is frozen to a commit at least five days old before it is proposed.

That default is not a wall, though. Freezing each input independently can produce a set that contradicts itself: one input resolves to a snapshot that has already removed a builder another input in the same set still calls. When a set like that refuses to eval or build, the window is loosened one day at a time — five days, then four, then three — re-frozen and re-tested at every rung, with the absolute-latest fallback held back as a last resort at the floor. A loosened window is disclosed in the pull request rather than going through quietly. A job that cannot be talked into absolute latest is a job worth having.

September — the product layer

September is where the fleet turned into something customers touch.

The client-facing service went from an interface to a deployed Go binary, database and routes, and then grew the parts a VPN product actually needs: a provisioning path, per-tier configuration, an upload/download fair-share shaper so no single peer can starve the rest, live per-peer telemetry, and a portal that ties it together. The WireGuard tiers were split so that the management plane and the data plane are not the same surface.

In the same month we published the things that are meant to be read: a logging policy, a privacy page and terms; an audit page generated at build time from signed host reports rather than prose; and an information-security management system whose documents live in the repository, where they can be reviewed like anything else. A no-log design was implemented at the level of the storage layer — counters that could be attributed to a single device were moved to a 30-day retention, per-peer read paths were removed from the admin surface, and mail submission moved inside the mesh so that handoff no longer leaves the network.

Backups moved to encrypted off-site object storage with a documented recovery runbook. A Kanboard instance, a Matrix homeserver, and the monitoring that watches all of it were deployed the same way everything else is: as files, through the pipeline, behind the health gate.

Where the machine-drafted work started

This is the part of the log that is genuinely unusual, so it is worth being precise about rather than sentimental.

The repository does not record which commits were drafted by a model, and no commit message says so. But commit style is a fingerprint. From the beginning of the window until the third week of July, every commit subject is written the way a person writes when nobody is watching: lowercase, abbreviated, terse. Then, in the week of 20 July, the shape changes.

Share of weekly commits authored under the human's own identity whose subject uses the conventional-commit form (type(scope):). Weekly commit counts are shown as n=.

The conventional-commit form appears at 0% through July, then jumps in the week of 20 July — the same week the site’s deploy pipeline and the reverse proxy were reworked — and settles at roughly 80% from the week of 10 August onward. That is the tell: the commit message changed shape weeks before the commit author did. Machine-drafted work written under a human identity looks exactly like that.

Four dates mark the transition:

19 July — the agent framework arrives as a flake input. Infrastructure, not authorship.
26 July — the first commits whose subjects use the conventional form land, under the human’s own git identity. Nobody is announced; the drafting style simply changes.
10 August — conventional-form subjects become the norm rather than a burst: 79% of that week’s commits, then 79% again the next week, then 88%. The author is still the human account.
29 August — the agent gets its own git identity. That week the split is 49% agent; the following weeks are 89%, 98%, 99%, 100%.

By September the fleet is being operated by an agent that opens pull requests against the same gates a person would have to pass, reads the failing job’s log when CI goes red, pushes a fresh branch, and merges only when the base-guard check says the branch is still mergeable. The 57% figure at the top of this post is a three-month average; the current rate is near total.

Think zen garden, not herd

The pipeline gets more of this post than the features do, and the reason is a background in systems engineering rather than in software. A production line is not a pile of machines; it is a set of controls, checkpoints and fail-safes arranged around a flow, designed so that an operator’s attention is spent on the exceptions and not on the routine. Years of that, plus rather a lot of Factorio and Satisfactory, leave you unable to look at a deployment process and see anything other than a line — one that is either balanced and instrumented, or being held together by hand.

So the fleet is built as a production line for configuration. Every input is inspected before it is admitted: flake inputs frozen to at least five days old, a window that loosens a day at a time rather than jumping to latest, a secret scan run before a branch can merge. Every unit is tested before it ships: the build gate, the staging deploy, the ephemeral per-pull-request preview. Every station has a checkpoint: the units that must be active before the switch is called good, and the baseline of long-running units. And nothing moves without a way back: the automatic rollback, the revert, the pinned revision. None of it is clever. All of it is deliberate.

The goal was never speed. It was set and forget — a stack this size (four hosts, a mesh, mail, monitoring, a client service, a portal, an audit pipeline) should be something you can walk away from, and the difference between a stack you can walk away from and one you cannot is almost entirely the quality of the line underneath it. Herding a behemoth by hand, every week, is exhausting and unreliable. Building the line so that the behemoth manages itself is calm, uniform and controlled — and it is the reason a three-month window like this one contains 1,221 commits instead of 1,221 incidents.

It is worth stating plainly, because the arcs above read like the output of a team: this is one person and one agent. The breadth is the point. A specialist goes deep in a narrow seam and can treat the rest of the line as someone else’s problem; a generalist has to hold the whole thing at once — the mesh, the mail path, the identity of the commit log, the retention periods, and the pipeline that ships all of it. That combination is rarer than it ought to be, and it is what makes a stack of this size tractable at a headcount of one.

Understanding every level of it — from the kernel up through the mesh, the services and the delivery line — and running the whole thing alone, without outside help, is a rarer combination than it should be. It is also one the industry is famously bad at noticing. That is its loss, not the line’s.

What this post does not claim

It does not claim the model is doing the thinking alone. The architecture, the decisions about what a VPN product should refuse to do, and the review of every change are the human’s — and the commit log shows it: the human identity still appears every week, opening and merging, and the corrections in the log are frequent enough to be a feature. What changed is who does the typing, and how much of the work survives the gates.

It also does not claim that volume is quality. 1,221 commits is a measure of throughput, not of correctness, and a fleet that could not be rebuilt from its repository would make the number meaningless. That is why the elastic measure in this post is not the commit count but the pipeline: thirteen workflows that decide whether any of those commits is allowed to reach a host.

What you can check

The configuration repository is public at git.ymrtech.com/ymrtech/nix-config. Every number above is a git log away.
The commit that started the pipeline is 073ce12; the agent’s first signed commits are from 29 August.
The audit page is generated from signed host reports and is checked by the build, not by this post.

Three DNS postures, one tunnel

2026-09-22T00:00:00-04:00

Every VPN sold as “three tiers” is quietly selling a bandwidth ladder: bronze is slow, silver is fine, gold is fast. Ours is not that. The tunnel is identical at every tier — same gateway, same shaping, up to 1 Gbit/s per connection on all three — and the price is the same $10/mo whichever you pick.

The only thing that changes is your DNS posture. That turns out to be the interesting part, because DNS names are the most revealing thing a network carries: they are unencrypted in the common case, they are written down by default, and they say what you were about to do before you did it.

3 tiers, one price

1 gateway, in Montréal

0 query logs — on any tier

pipe — the tunnel, and nothing else

The pipe configuration carries no DNS setting at all. Your device keeps the resolver it already runs, so lookups leave the tunnel and go to your ISP, your router, your browser’s built-in resolver — whoever was already answering.

What changes: your traffic is tunnelled; your DNS is not.
Who sees your lookups: exactly who saw them before you subscribed.
Who does not: us. We are not in the path, and we do not learn the names.

That is the shape of it: pipe is the tier where you hand us the least. It is also the tier where a browser doing its own DNS-over-HTTPS can silently bypass your expectations of the tunnel, because the device still owns the decision.

privacy — our resolver, inside the tunnel

privacy puts a resolver at 172.16.40.1 — an address reachable only from inside the tunnel — and hands you that as your DNS server. Lookups go up the tunnel to it, and it forwards upstream over DNS-over-TLS, so the last leg is encrypted too.

What changes: your lookups stop being visible to the network you are on.
Who sees your lookups: us, in the sense that our resolver answers them — and this is the part that matters, so read the next paragraph rather than the bullet.
What we do with them: the resolver runs with its query log disabled and its statistics disabled. No name in a log, no counts to aggregate. That is stated with the rest of the retention list on what we log and re-asserted by the daily signed audit rather than by prose.

This is the tier where “we keep no logs” stops being a claim about our good intentions and becomes something with machinery behind it: a check re-applies the setting, sends a real query through the resolver, injects a canary name that cannot occur by accident, and then requires that nothing was written. If a log appears, the check fails and a failing report is still signed and published at its URL before the unit fails — a violation is a published violation, not a quietly deleted one. The full mechanism, and how to fetch and verify it yourself, is on the audit page.

security — the same resolver, with a filter in front

security is privacy plus AdGuard Home at 172.16.41.1, with filter lists that block ads, trackers and known-malware domains before your device ever connects to them.

What changes: a large part of the web’s background noise never resolves.
Who sees your lookups: the same answer as privacy — the resolver inside the tunnel, without query logs or statistics.
What it costs you: a filter is an opinion. A name on a list is a site that does not open, and the list is not always right. If a page fails to load and you cannot explain why, this tier is the first thing to rule out — switch that one device to privacy and try again.

Where the tier is chosen

Per device. One subscription runs a laptop on privacy, a phone on security and a work machine on pipe, three devices per account, each with its own assignment in the portal. You are not choosing a plan; you are choosing how much of your DNS to hand us, one device at a time.

Checking that it is actually in force

Do not take the tier name on a portal page as evidence.

The portal includes a DNS leak test that mints a unique probe name, has your browser look it up, and then checks whether that lookup reached our resolver inside the tunnel. A name invented for that one test cannot be in any cache and cannot be in any list, so where it shows up is a real answer — including when the answer is “it did not arrive here”.

If you want to check by hand instead: pipe leaves your configured resolver alone, privacy and security point it at a 172.16.x.x address that is only reachable while the tunnel is up. The technical dossier documents the resolver mechanics in full.

What no tier does

Being straight about the edges:

It is not a geo-unblocking service. One gateway, one exit, no rotating addresses, no residential proxies. Your traffic exits in Canada, which is a feature if that is where you want it to exit, and not one if you were hoping to look like you are elsewhere.
DNS is not the whole story. A site you log in to still knows who you are. A browser can still be fingerprinted. security blocks names, not malware that is already on your disk. The list of things a tunnel cannot fix is on what we log, and it is a real list.
pipe is a legitimate choice. If you already run your own resolver and you trust it more than a provider you have never met, that is the tier built for you, and it is not a downgrade.

The audit page is built from signatures, not sentences

2026-09-21T00:00:00-04:00

Every VPN provider has a page that says it keeps no logs. The page is prose. You either believe it or you do not, and there is no third option, because there is nothing on it to check.

Our audit page is not prose. It is generated at build time — before Jekyll renders a single template — from machine-readable reports that the hosts publish and sign themselves. That difference is the whole post, so here is exactly how it works.

What the page is made of

Each host runs a scheduled job that checks the things we claim about it: that the no-log invariants hold, that the resolver settings are what they are supposed to be, that the unit files have not drifted. It writes a JSON report, signs it with a detached ed25519 signature, and publishes the report, the signature, and the accepted signer alongside each other.

At build time, tools/audit-report.rb fetches those artifacts over HTTPS and verifies each signature with stock ssh-keygen -Y verify against the signer key pinned in this site’s repository (_data/audit-pins.json). Only then does it substitute real numbers into the page you are reading.

No YMRTECH software is involved in the verification step. That is deliberate: a verification you have to trust our code to perform is not much better than a sentence. ssh-keygen is on your machine already.

What the build refuses to do

The gate is not decorative. The build stops, and the deploy does not happen, if:

a signature does not verify — the artifact is not the bytes that were signed;
a key rotates without being announced — a new signer is not accepted just because it signed something; the pin has to change in a reviewed commit first;
a report claims to be a host it is not — a valid signature from mail does not authorize a report about vpn;
a signed index disagrees with the file it describes — a report whose hash is not the hash the index recorded is a rewritten report.

There is one case that is deliberately not a failure: a host that is simply unreachable. The page records it as unavailable and says so, because the alternative — rendering a green box for a host nobody could reach — is exactly the lie the rest of this design exists to prevent.

Why the failure is still published

The most interesting property of the whole thing is what happens when a check fails.

The job writes and signs the report first, and only then fails. A failing run therefore leaves a valid, signed, published report at its URL saying that something is wrong. Deleting the evidence of a failed check is not an option the machinery offers, and that was the design goal: the moment when you would most want to quietly tidy up is the moment the artifact is already out there.

Why month-long claims are still claims you can check

“Nothing in the log” on the day you visit the page is a small statement. The month-long version is a bigger one, and it gets its own machinery: a monthly roll-up recounts every dated daily report in the period and re-verifies each signature, and a closed month is frozen once published with its SHA-256 recorded in the index. A month that has already shipped cannot be revised later without the index disagreeing with it — and the index disagreeing is one of the conditions that fails this build.

What a signature does not prove

A signature proves origin, not correctness. It proves a named host produced these bytes and that nobody changed them afterwards. It does not prove the host was telling the truth, and it does not make us independently audited.

These reports are produced by the infrastructure they describe. We hold no third-party audit, no accredited certification, no SOC 2 report, and we do not claim one — the standards page is explicit about which words we are allowed to use. And the audit’s own generator is not public source: the boundary around what is public, and what that costs you, is written out at the public-source boundary, on the same page. That section exists because a page full of cryptography invites exactly the wrong conclusion, and the wrong conclusion is the one thing this is not allowed to produce.

The rest of the public posture — what we log, what we do not, and how the site’s own stack is built — is on what we log, standards and the technical dossier.

Staging is not production: how this site ships

2026-09-20T00:00:00-04:00

This website is built by a pipeline you can read: the workflow is .forgejo/workflows/ymrtech-deploy.yml in the same public repository as the pages themselves. It is short enough to read in one sitting, and it contains one design decision worth writing about, plus one incident that produced it.

Three triggers, two environments

Event	What runs	What it writes
Pull request	the full gate, plus a throwaway preview of the branch	nothing outside the preview root
Push to `main`	the full gate again	staging only
Manual dispatch with promote to production	the full gate a third time	production

main is the release branch for both environments. They differ by trigger, not by branch: merging to main automatically verifies the exact commit that will later go to production, and production only moves when a human dispatches the promotion, which re-runs the entire suite before touching the prod docroot.

The gates are the same in every job — icon font coverage, signed audit artifacts, the Jekyll build, a check that the contact form is actually visible in the rendered page rather than hiding in an HTML comment, htmlproofer over the built site, a fragment-resolution check, the palette check, and the writing-content check that guards these posts. Each job is self-contained, so a gate added in one place has to be added in all four. That is a cost, and it is paid deliberately: a job that depends on another job’s artifacts can be skipped into a false green.

The day a pull request overwrote staging

The workflow’s own comments keep the scar tissue. Originally there was a single job, and it ran its rsync on pull_request events too. The consequence: opening or updating a PR — any PR, from any branch — replaced /srv/ymrtech-stage with that branch’s build. Staging stopped meaning “what main verified” and started meaning “whatever was pushed most recently”, which is the same as meaning nothing.

The fix is one condition on the deploy job — staging is gated to push and explicit workflow_dispatch, and the cleanup that removes a preview is gated to pull_request events instead. That one line is the difference between a staging environment and a scratch directory.

Previews get a path, not a hostname

Rather than give each branch a subdomain, a PR gets a path: stage.ymrtech.com/preview/pr-<N>/, built with --baseurl /preview/pr-<N>. That works because every internal link and asset reference in this site is generated through a Jekyll filter that applies the site’s baseurl — relative_url for links, asset_url for the handful of design assets, which adds the file’s own content hash on top. There is nowhere to hardcode a root-relative path and have it render, which is exactly the kind of property you fail to maintain until the day you need it.

The preview root is a separate tmpfs from the staging docroot, so a staging deploy (rsync --delete) cannot erase a preview, and a preview cannot overwrite staging. It is deleted when the PR closes. And because it is unreviewed branch content published on a real domain, the pipeline verifies the served URL after publishing it: it polls for HTTP 200 and then requires the X-Robots-Tag: noindex and Cache-Control: no-store headers to be present. If those headers are missing, the job fails — the assumption being that a response without them is not the preview route that was supposed to answer.

The bits that only show up in production

Two failures are worth keeping in mind as arguments for testing the real artifact rather than the source:

327 missing files. The first real preview run failed html-proofer with 327 “does not exist” errors, because the site was built with a /preview/pr-<N> baseurl while the proofer resolves internal references against the docroot on disk, where no such directory exists. The fix is --swap-urls, and the workflow now documents why it takes two replacements rather than one: with a single pair the prefix is rewritten to /, producing protocol-relative URLs like //assets/x, which the proofer also rejects. All three variants — no swap, one pair, two pairs — are reproduced in the comment, along with which one is clean.
A token that cannot be baked in. The contact endpoint rejects unauthenticated posts, so the form needs a token — but this repository is public and a token in the source is a token you cannot rotate without a rebuild. The deploy writes it from a host-readable secret file into a same-origin JSON endpoint at deploy time, and if that file is missing the endpoint stays locked and the form offers a mailto fallback instead of failing silently.

Why this is here rather than in a wiki

Because it is the same argument as the audit page: a delivery claim you cannot inspect is a claim about intentions. The workflow is public, the failure modes are documented next to the code that caused them, and the pipeline that builds this page is the one that would have caught me getting it wrong — the gate for these posts runs in all four jobs, and it fails the build if a post’s stable slug and its rendered campaign tag stop agreeing.

For the infrastructure underneath it — the hosts, the runner, the deploy health gate and the automatic rollback — see how the stack is built.

What "no logs" can and cannot promise

2026-09-19T00:00:00-04:00

“No logs” is the most repeated sentence in this industry and the least defined. Every provider says it; almost none of them tell you what the exceptions are, and a policy that hides a meter is worse than one that explains it.

So this is the whole list — the one already published, with the reasoning attached, on what we log.

What is not recorded

Browsing history. Not recorded.
The DNS names you resolve. Not recorded — on any tier. The resolver runs with its query log disabled and its statistics disabled.
Traffic content. Encrypted end to end. We carry it and cannot read it.
Your ISP-assigned address, in connection with your account.
Connection times. No record of when you connected or disconnected.
Your device’s real network.

What is recorded, and why

Four rows. Not because four is a nice number, but because each one is something the service cannot run without:

Data	Why it exists	Retention
Email address, password hash	your account	life of the account
Subscription and payment record	billing; card details are Stripe’s and never reach us	life of the account
Device records — name, tier, tunnel IP, your public key	to provision and revoke access. Configuration, not activity	life of the account
Aggregate data-volume counters	to keep the service fast and fair, and to stop abuse. Byte counts only: no destinations, no DNS, no browsing	currently up to 30 days, per device

Plus one that arrived recently and is worth naming because it is unusual: the speed test results you run yourself from the portal — two rates, byte counts, latency, and which of your own devices you labelled the test with. The five newest per account, pruned as new ones arrive. It is the result that was on your screen and nothing about the transfer behind it: no address at either end, no destination.

On the counters. The counters show how much you transferred, not what. A byte total cannot be turned back into a browsing history, and it is never combined with destination data because there is no destination data to combine it with. It is disclosed for the same reason every row is: a meter you are not told about is a meter you cannot argue with.

Server logs. Ordinary operating-system logs — service starts and stops, crashes, errors, deployments. Kept for 14 days, asserted by the automated audit to contain no customer traffic or DNS data and no client-tier addresses.

How the claim is tested rather than asserted

This is the part that separates a policy from a promise:

A real query is sent, then the system checks that nothing was written. A DNS query goes through each resolver; the check then requires that no log appeared. A canary name — one that cannot occur by accident — is injected for the same purpose, so “we found nothing” cannot be an artifact of looking in the wrong place.
The result is signed and published, daily. Each host signs a machine-readable report with a detached ed25519 signature; the site’s build verifies those signatures with stock ssh-keygen and refuses to render if one fails. The mechanism, and how to fetch and check it yourself, is on the audit page.
A failure is published, not hidden. A failing check still writes and signs the report first, then the unit fails — so a violated report exists at its URL and stays there.
The infrastructure is declarative and version-controlled. Nothing is hand-edited on a running machine; what is published is what runs.

The warrant canary is not published yet

The policy commits to a signed, periodically renewed warrant canary at a stable address, recording how many disclosure demands have been received. It is not published yet. Saying so is the point: a commitment listed next to things that are already true, without a marker for which is which, is how “we’ve added X” quietly becomes “we said we have X”. When it exists, it will be linked from what we log; until then, treat it as pending.

What a tunnel cannot protect you from

What a tunnel does not cover, in full:

malware on your own device — a tunnel is not an antivirus;
logging in to an account that knows who you are — Google still knows which Google account you used;
a browser using its own DNS in a way your device does not route through the tunnel. This is a real bypass, which is why the privacy and security tiers put the resolver inside the tunnel and why the portal includes a leak test that reports where a freshly minted name actually went;
anything you do after you disconnect.

A VPN hides your traffic from the network you are on. It does not hide you from the services you deliberately log in to, and any provider that implies otherwise is selling you a feeling.

Tofu boxes: why the icons ship as ~6 KiB of font

2026-09-18T00:00:00-04:00

Update (2026-10-16). The icons no longer ship as a font. Every one is now an inline SVG drawn from a registry in _data/icons.yml, which reaches the outcome this note argued for and could not yet get to: an inline SVG has no code point that can fail to map, so the tofu box it is about cannot occur. The font, the subset generator, the manifest and the coverage gate are gone. This note is left as written — it is the record of why a subset was the right call then, and of how the argument turned.

The icons on this site are not images and they are not an icon library. They are Nerd Font glyphs — private-use code points such as U+F413 and U+F08C8, drawn in the same monospace font as the text.

That is a nice trick right up until you remember the obvious: no stock system font maps those code points. A visitor who does not happen to have a Nerd Font installed gets a tofu box — the hollow rectangle of a missing glyph — in front of every link in the navigation. That is the failure the in-house font exists to prevent.

The fix, and the version of it I did not want

The lazy fix is a <link> to a third-party font CDN. That is a runtime dependency on someone else’s uptime, one more supply-chain hop on a page whose whole argument is that you should be able to check things, and a reference that can stop resolving without a single commit in this repository to warn you.

So the font is served from this origin, from assets/fonts/:

Upstream: ryanoasis/nerd-fonts, with the release tag pinned in tools/nerd-font-subset.py;
Family: JetBrainsMono Nerd Font Mono, Regular and Bold;
Subset: only the icon and graphic code points this repository actually uses;
Size: roughly 6 KiB per weight;
Licence: OFL.txt, verbatim from upstream — SIL Open Font License 1.1.

Six kilobytes is small enough to inline, which is a nice place to land: the page carries its own icons, so drawing one is a local file read — nothing about the glyphs depends on a host we do not control, and nothing changes under us when somebody else’s URL does.

The generated files are generated

tools/nerd-font-subset.py scans the repository for glyphs in the icon ranges — arrows, box drawing, block elements, geometric shapes, and the two Nerd Font private-use blocks — subsets the pinned upstream release down to exactly those glyphs, writes the two woff2 files, deletes the ones it superseded, and rewrites the generated @font-face block in assets/css/style.css in place.

Two details in that script are load-bearing:

Latin text and general punctuation are deliberately not in the subset. Those code points keep falling through the font stack to Fira Code and then to the platform monospace font, exactly as they did before. Adding a 300 KiB text font to fix 6 KiB of icons would have been a different decision with a different cost.
The file name embeds a content hash. A stylesheet is referenced with a version query, and the font file it points at is named after its own bytes, so a cached copy of an old stylesheet can never pin a stale subset. The generated files are committed together with the stylesheet change that references them.

The failure is silent, so there is a gate

This is the kind of bug that returns without warning. A missing glyph does not produce a 404 — the font loads fine, and then draws a box. A glyph dropped from the stylesheet’s coverage looks exactly like a glyph nobody ever wrote.

So coverage is asserted at build time, in all four CI jobs:

tools/icon-codepoints.txt records exactly which code-point spans the shipped subset covers.
tools/check-icon-coverage.rb re-scans the sources on every build and fails if the site uses a glyph the subset does not cover — or if style.css references a font file that is not in the repository.

The consequence is the property worth having: adding a new icon without regenerating the subset is a red build, not a tofu box in production. The mistake is now louder than the fix.

The same reasoning runs through the rest of the theme — colour is checked against declared tokens, so the palette cannot drift out of the dark/light switch either. See how the stack is built for the rest of the pipeline, and the audit page for the version of this argument that concerns signatures rather than glyphs.