Security · Investigations · AIActive incident? Request help

Technical commentary. Code and historical experiments illustrate methods; they do not establish a result for your organization.

Field Notes · Security Architecture · 6 min read

A standard survives independent teams when it fails closed, signs its interfaces, and writes its decisions down.

The problem is not writing the standard

Standards usually arrive as documents. A team I do not manage reads the document, agrees with it, and then ships whatever its tooling lets it ship. Some time later the standard describes a system nobody runs. The failure is not disagreement. It is that the document had no mechanism. A standard survives independent teams when a machine checks it, when the check cannot be quietly turned off, and when the people who adopted it can read why it exists without asking the author.

Rule one: fail closed

The first property is that the tool enforcing the standard refuses when it cannot verify. Tasia, my open-source reviewer for private-AI deployment configuration, runs in CI as a gate with a severity threshold. A configuration file it cannot parse produces Decision: ERROR and a distinct non-zero exit code reserved for tool and configuration errors, never a misleading PASS. The README states the rule in one sentence: a gate that cannot read your stack will not vouch for it.

assay, the multi-model validation harness, applies the same rule at the network boundary. Every outbound request passes a signed scope gate. A scope document that is missing, malformed, expired, or unverifiable produces zero requests and the error exit code. The gate cannot be disabled in CI. The development override demands an explicit environment variable, prints a banner, and marks every decision it made with a key id that says so. The trust store that verifies tool manifests behaves the same way: a verifier with no trusted keys refuses to run.

Fail closed matters for adoption because it removes the option of partial compliance. A team either passes the check or it sees a red build with a reason code. There is no state where the tool ran, could not tell, and let the push through. I wrote about the forensic version of this rule in Forensic Parsers Should Fail Closed; the engineering version is the same principle pointed at your own pipeline.

Rule two: sign the interfaces

The second property is that the interfaces other teams depend on carry provenance a machine can check. In assay, every tool manifest carries a signer id, a signing time, and an Ed25519 signature computed over the canonical JSON bytes of the manifest (RFC 8785) with the signature field cleared. The trust store is a directory of public-key files. Verification returns one of a closed set of reason codes: unsigned, signature invalid, unknown signer, or provenance valid. The verify command exits non-zero on any failure, so CI gates on it.

Two design choices make this adoptable by teams I do not manage. First, the signature is provenance, not authorization. It proves the manifest came from a trusted key and has not changed since. What the manifest is allowed to do is decided separately by the policy engine, which takes the verification result as its first input and denies on anything but a valid signature. Keeping those questions apart means a team can rotate keys without rewriting policy, and change policy without re-signing manifests. Second, key rotation is operationally boring: add the new public key file, re-sign, verify, delete the old file. Manifests signed by a removed key fail as UNKNOWN_SIGNER, which is the intended outcome.

Canonical JSON is the quiet part that makes this portable. The ADR that introduced it records why: a hash is only useful if every party computes it over the same bytes, and a language's default JSON encoder is not a specification. assay implements RFC 8785 in the repository and refuses non-integer numbers, so an implementation in any language can recompute every digest from the RFC alone.

Rule three: a results contract

The third property is that a result has a fixed shape and a fixed home. assay's results contract says one run is one directory under results/, holding the run summary, the redacted findings, the overlap matrix, the costs, and a ci.json that names the CI run that produced it. Derived files are pure functions of the primary files, and assay overlap --check recomputes the overlap and exits non-zero on any byte difference. Only runs with at least one real model provider are committed; fake-only pipeline checks stay in workflow artifacts.

The committed run 20261007-215749-7290ba88 is a small example of the contract working. Its run.json records the CI run URL, the git SHA, the audit head hash and record count, and a mode field that reads degraded because a configured provider had no key. The report says so in its first line. A results contract that publishes its own degraded state is more useful to another team than a clean number with no provenance, because the team can decide what the number is worth.

Rule four: the ADR is the mechanism

The fourth property is that the decisions are written as architecture decision records: status, context, decision, consequences. assay's ADR series covers the signed manifests, the audit log, zero data retention, the provider router, the finding key, the fail-closed gate, refusals, validation states, attack-path chaining, the results contract, and continuous mode.

The reason ADRs survive independent teams is the consequences section. ADR 0004 on the audit log states plainly that removing the last line of the log is not detectable from the file alone, and that the run report commits the head hash and record count for that purpose. ADR 0010 states that finding classes without a checker can only ever be theorized, and calls that a visible gap rather than a silent overclaim. A team adopting a standard needs to know what it costs and where it stops. An ADR that admits its limits is one a team can argue with, extend, or decline on the record. A standard that only lists benefits gets agreed to and ignored.

What this does not establish

Conclusion

Standards that survive independent teams are enforced by tools that fail closed, carry signatures a machine checks, publish results under a contract that regenerates, and record their decisions with the costs stated. Write the standard as a mechanism and the document becomes a description of something that is running. Write it only as a document and it becomes a description of something that used to be intended.

Related: A Technical Claim Is Not Evidence, on why a stated property is a hypothesis until something checks it, and Agents Act Only Through Signed Tools, on the execution boundary these standards protect.

Read the mechanisms, not the summary

The architecture decision records described here are public in assay's docs/adr directory, alongside the trust store, the policy file, and the committed run. The fail-closed configuration gate is Tasia. If you adopt any of it, the consequences sections are the part to read first.

Know a platform team trying to get a standard adopted outside its own org chart? This note is for them.

← All Field Notes

Apply this to your decision.

Start with an Advisory Hour at $500/hour, or review the current service scopes and fees.

Book an Advisory Hour — $500Services and prices