The problem
My systems change underneath the documents that describe them. In June a model-written review of my projects rated every one of them “Excellent”. A second review that actually ran git status and listed files found a document describing a deleted repository as live, and a project built around a record file that didn’t exist. Reading prose misses drift. Checks that run catch it.
What it does
There are three pieces.
The résumé check runs weekly. It searches my résumés and cover letters for phrases that describe systems no longer running, and it re-derives each number I still claim (records in the index, photos, passages, Pulse’s file size, test counts) from the live machine. If a claim no longer holds, the check fails and the résumé gets fixed. The check stays as written.
vm-audit runs every morning: a read-only audit of the whole server across 11 areas, including open ports, running processes, scheduled jobs, dependencies, permissions on secret files, unpushed git work and outdated model names in code. It keeps a ledger of findings and reports only what is new, changed or resolved since the last run. A quiet day costs no model tokens.
doc-integrity lets any document carry its own assertions in a small grammar: this file exists, this service answers, this path is gone. A fast, filesystem-only pass runs every time a coding session opens on the server and again inside the morning audit. The full pass, which also checks services and git, runs when I ask for it.
How it’s built
vm-audit and doc-integrity use only Python’s standard library, so no outside package can break underneath them. The résumé check is a shell script. In vm-audit, scripts collect and compare. A model reads only the day’s changes and writes a short summary for me. The ledger is written by code alone. Model output is prose for a person and never feeds back into state, so a hostile string in a log file can’t change what the audit believes. If the model call fails, a plain report goes out instead. A canary run pushes a synthetic finding through the whole loop to show the loop still works.
Decisions
- Detect and report, never fix. An early automatic process killer, written to stop a forbidden program, killed the terminal that was writing it, because that command line contained the word it was hunting for.
- Report what changed. The same warnings reported every morning become invisible, so the ledger remembers what it has already said.
- A check gets tested by breaking its input. A check I’ve only seen pass can’t be told apart from one that can’t fail, so each condition gets one deliberate break to confirm it fires.
- Dead claims move to the past tense. “Built and ran a retrieval system over about 509,000 records, since decommissioned” survives being asked about, and “live” doesn’t.
How it broke, and what changed
A résumé of mine described a retrieval system over about 509,000 records as live. It had been decommissioned 55 days earlier. The database and its vector extension still existed, which is why nobody noticed, but every table was empty. That résumé had already gone out with job applications. A second line claimed local model inference on a service removed that same day.
The résumé was the one kind of document here with no freshness check, and it is the one that goes to strangers. The résumé check was written the same day. Its first version read only the Markdown files and missed their plain-text copies, so it reported clean on a file still carrying six false claims. That was caught within the hour, and it is why checks here now get run against a known-bad input before anyone trusts them. The replacement numbers were larger than the false ones: the live index holds 763,186 records.
What’s still rough
- The checks cover what someone thought to write a check for. A document that contradicts the code in meaning, as opposed to naming a dead path, is left to a weekly model read, which is the weaker kind of review.
- doc-integrity currently reports 96 warnings, 81 of them references to paths that no longer exist. That many warnings is noise, and noise gets ignored.
- If the server itself goes down, the only signal is a missing morning email. An outside dead-man’s switch is supported and not yet switched on.
- The numbers from my personal records can only be re-derived on my server, so a reader has to take the check’s word for them.