Mirrored lectures 6a/6b: the authenticated structure, drawn and domain-separated

The trust architecture has exactly two roles and the curriculum now
mirrors that split structurally - the conceptual burden is the design,
stated as such to the student:

- 06a THE PROVIDER'S SIDE (singleton). Domain banner in the provider's
  voice. The full build pipeline run live in a scratch log made from
  the REAL attestations: verify (Lean replay = the leaf-making step,
  the only expensive one - the shipped evidence IS its output) -> leaf
  (0x00 domain separation) -> tree -> STH signed via the MERKLEIZED
  LIBRARY -> the self-inclusion check embedded in the signature block.
  A generated SVG draws the student's own tree: leaves, internal
  nodes, root, and the signature box, framed in the provider's domain
  color. Closes with the singleton-vs-many justification table
  (key/cost/obligation asymmetry) and exercises.
- 06b THE AGENT'S SIDE (one of many). Domain banner in the agent's
  voice: you own the public key, the evidence files, ~25 lines of
  hashing - and explicitly NO Lean. The COMPLETE RFC 9162 inclusion
  verifier is implemented from scratch in one cell (hashlib only, no
  pacta imports for the core) and run against the REAL dalek receipt
  (leaf 4 of 8, three siblings, dogfood-signed root); then the STH
  signature, the provider's signing_provenance read and interpreted
  (why the agent still re-checks inclusion itself), the pin store, and
  an SVG of the real log with the agent's path highlighted against the
  grey leaves it never needs. Cost line: ~4 hashes + 1 signature.
- Lecture 06 now routes students into the pair and states the mirror
  rule ("if you cannot say which notebook a step belongs to, you have
  not understood the step"); lecture 09 records that dogfood now runs
  in BOTH directions; course map + README updated.

Every cell of 06a/06b/09 executed against the real evidence before
commit (SVGs render in Jupyter, fail soft in plain exec). One
generation bug found and fixed: a single-backslash \\x00 in the
generator produced a literal NUL byte in a cell. 50/50 tests green
with the notebook inventory at twelve.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
mrwulf 2026-07-06 15:26:32 +02:00
parent b8ffbafa7f
commit 19d25458e8
7 changed files with 963 additions and 2 deletions

View file

@ -64,6 +64,7 @@ The `notebooks/` directory contains a zero-to-hero teaching sequence for undergr
- `04_proof_hygiene_and_boundaries.ipynb`: `sorry`, local axioms, trivial targets, manifest coverage.
- `05_third_party_attestation_provider.ipynb`: provider trust transformation and signed attestations.
- `06_merkle_transparency_logs.ipynb`: RFC 9162-style Merkle proofs, STHs, Ed25519/ML-DSA policy.
- `06a_provider_build_the_log.ipynb` / `06b_agent_verify_inclusion.ipynb`: the MIRRORED PAIR - one provider (builds, Lean-verifies, signs with the merkleized library, self-checks its own inclusion), many agents (verify inclusion from scratch in ~25 lines, no Lean); the domain separation is the design and the lecture structure mirrors it.
- `07_agent_consequences.ipynb`: receipt-gated artifact builds and wallet-denial policy.
- `08_capstone_research_program.ipynb`: audit the shipped R4 evidence; design the R5 discharge plan.
- `09_dogfood_verified_crypto.ipynb`: the proven-path verifier in the agent's own loop; hybrid-PQC posture.

View file

@ -95,7 +95,16 @@
"5. `05_third_party_attestation_provider.ipynb`\n",
" Learn how a proof-checking service can transform hard local verification into provider trust.\n",
"\n",
"6. `06_merkle_transparency_logs.ipynb`\n",
"6. `06_merkle_transparency_logs.ipynb`, then the MIRRORED PAIR\n",
" `06a_provider_build_the_log.ipynb` / `06b_agent_verify_inclusion.ipynb`\n",
"\n",
" The trust architecture has exactly two domains - ONE provider\n",
" who builds and signs the authenticated structure (and pays the\n",
" Lean bill), MANY agents who verify inclusion proofs in\n",
" milliseconds. The course mirrors that split structurally: 6a is\n",
" written entirely in the provider's voice, 6b entirely in the\n",
" agent's. If you cannot say which notebook a step belongs to,\n",
" you have not understood the step.\n",
" Build the Merkle accumulator intuition behind inclusion proofs, consistency proofs, and Signed Tree Heads.\n",
"\n",
"7. `07_agent_consequences.ipynb`\n",

View file

@ -160,6 +160,18 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Two domains, two notebooks - by design\n",
"\n",
"Everything above is the shared VOCABULARY. The system itself has\n",
"exactly two roles, and the next two notebooks separate them on\n",
"purpose: **6a - the provider** (a singleton: builds every leaf via\n",
"Lean replay, builds the tree, signs the root with the merkleized\n",
"library, and Merkle-verifies its own signing library's leaf before\n",
"signing), and **6b - the agent** (one of many: the provider's\n",
"public key, the evidence files, ~25 lines of hashing, and nothing\n",
"else - explicitly NO Lean). Keep the mirror in mind as you drill\n",
"the primitives below; each drill belongs to one side.\n",
"\n",
"## Split Views: why a receipt is not enough\n",
"\n",
"Everything above verifies ONE receipt against ONE signed tree head. A malicious provider can maintain TWO trees - one shown to you, one shown to the world - and both views verify perfectly in isolation. This is EQUIVOCATION, and the defense is memory: pin every tree head you accept, and demand that every later tree head be CONSISTENT with your pin (same size -> same root; larger size -> a verified consistency proof from your pinned size; smaller size -> rollback, reject forever).\n",

View file

@ -0,0 +1,256 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Lecture 6a: THE PROVIDER'S SIDE - Building the Authenticated Structure\n",
"\n",
"> **DOMAIN BANNER - read this first.** In this notebook YOU ARE THE\n",
"> PROVIDER. There is exactly **one** of you per log. You hold the\n",
"> signing key. You own a Lean toolchain and hours of compute. You\n",
"> carry the append-only obligations. Nothing in this notebook is\n",
"> ever executed by an agent - and that asymmetry is not an\n",
"> implementation detail, it is the entire design (see the\n",
"> justification at the end).\n",
"\n",
"The provider's job, end to end: **verify -> leaf -> tree -> sign ->\n",
"self-check**. Only the first step involves Lean; everything after\n",
"is hashing and one signature.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Learning Objectives\n",
"\n",
"- Build the full authenticated data structure from real attestations: leaves, tree, Signed Tree Head.\n",
"- Place the Lean verification correctly: it is the LEAF-MAKING step, the only expensive one, and it never travels to the agent.\n",
"- Sign the root with the merkleized library and run the provider's own inclusion self-check (\"the provider eats its own dogfood\").\n",
"- Justify the singleton/many split as a design decision.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 1 - Verify (the expensive step, done ONCE)\n",
"\n",
"The leaf content is a signed **attestation**: the outcome of replaying\n",
"every Lean proof of one repository under lean-guard (~30 minutes of\n",
"kernel re-checking per fork on the reference machine). This notebook\n",
"does NOT re-run that - the shipped `evidence/` attestations ARE that\n",
"step's output. What matters architecturally: **the Lean cost lives\n",
"here and only here.** No agent ever pays it again.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import sys\n",
"\n",
"repo_root = Path.cwd()\n",
"if not (repo_root / \"src\" / \"pacta\").exists():\n",
" repo_root = repo_root.parent\n",
"sys.path.insert(0, str(repo_root / \"src\"))\n",
"sys.path.insert(0, str(repo_root / \"provider\" / \"src\"))\n",
"\n",
"from pacta.yamlio import load_data\n",
"\n",
"attestations = {\n",
" fork: load_data(repo_root / \"evidence\" / f\"{fork}-ed25519.attestation.yaml\")\n",
" for fork in [\"dalek\", \"anza\", \"risc0\", \"betrusted\"]\n",
"}\n",
"for fork, att in attestations.items():\n",
" certs = att[\"certificates\"]\n",
" clean = sum(1 for c in certs if c[\"status\"] == \"proven\" and c[\"axiom_status\"] == \"clean\")\n",
" print(f\"{fork}: {clean}/{len(certs)} proven | commit {att['subject']['repo_commit'][:8]} | guard: {att['machine_protection']['lean_guard'].rsplit('/',1)[-1]}\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Steps 2+3 - Leaf and tree (cheap, mechanical)\n",
"\n",
"Each attestation is wrapped, canonically serialized, and hashed with\n",
"the RFC 9162 leaf prefix `0x00`; pairs of nodes hash with prefix\n",
"`0x01`. Build a REAL provider log in a scratch directory - you are\n",
"the provider, so mint your own key first:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import tempfile\n",
"from pacta.signing import generate_ed25519_keypair\n",
"from pacta_provider.transparency_log import TransparencyLog\n",
"\n",
"state = Path(tempfile.mkdtemp(prefix=\"provider-lecture-\"))\n",
"generate_ed25519_keypair(state / \"provider.key\", state / \"provider.pub\")\n",
"log = TransparencyLog(state / \"log\")\n",
"log.init(\"lecture-provider\", state / \"provider.pub\")\n",
"\n",
"receipts = {}\n",
"for fork, att in attestations.items():\n",
" att_path = state / f\"{fork}.attestation.yaml\"\n",
" from pacta.yamlio import dump_data\n",
" dump_data(att, att_path)\n",
" receipts[fork] = log.append_attestation(att_path, state / \"provider.key\", state / \"provider.pub\")\n",
"print(\"tree size:\", receipts[\"betrusted\"][\"tree_size\"])\n",
"print(\"root:\", receipts[\"betrusted\"][\"sth\"][\"root_hash\"][:32], \"\u2026\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Steps 4+5 - Sign the root, then CHECK YOURSELF\n",
"\n",
"The tree head is signed with the **merkleized library itself** (when\n",
"the dogfood binary is built): the Ed25519 code that signs this root\n",
"is the same pinned dalek source whose proof attestation is a leaf of\n",
"this very tree. Before signing, the provider runs the SAME Merkle\n",
"inclusion verification an agent would run - on its own signing\n",
"library's leaf, against the tree it is about to sign - and embeds\n",
"the verdict in the signature block. A root signature that names the\n",
"leaf vouching for the code that produced it:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import json\n",
"\n",
"sth = receipts[\"betrusted\"][\"sth\"]\n",
"ed = sth[\"signatures\"][\"ed25519\"]\n",
"print(\"signing backend:\", ed.get(\"signing_backend\"))\n",
"print(json.dumps(ed.get(\"signing_provenance\", {\"note\": \"dogfood binary not built on this host - OpenSSL fallback, provenance omitted\"}), indent=1))\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## The structure you just built, drawn from your own log\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from pacta.transparency import leaf_hash, merkle_root, node_hash\n",
"\n",
"def svg_merkle(leaf_hashes, highlight=None, title=\"\", domain=\"PROVIDER: builds every box below\", color=\"#1e7f4f\"):\n",
" n = len(leaf_hashes)\n",
" width, lh, lv = 980, 108, 92\n",
" levels = []\n",
" level = [bytes.fromhex(h) if isinstance(h, str) else h for h in leaf_hashes]\n",
" levels.append(level)\n",
" while len(level) > 1:\n",
" nxt = []\n",
" for i in range(0, len(level) - 1, 2):\n",
" nxt.append(node_hash(level[i], level[i + 1]))\n",
" if len(level) % 2:\n",
" nxt.append(level[-1])\n",
" levels.append(nxt)\n",
" level = nxt\n",
" height = 130 + lv * len(levels) + 60\n",
" out = [f'<svg xmlns=\"http://www.w3.org/2000/svg\" width=\"{width}\" height=\"{height}\" font-family=\"monospace\" font-size=\"11\">']\n",
" out.append(f'<rect x=\"4\" y=\"4\" width=\"{width-8}\" height=\"{height-8}\" fill=\"none\" stroke=\"{color}\" stroke-width=\"2\" rx=\"8\"/>')\n",
" out.append(f'<text x=\"16\" y=\"24\" fill=\"{color}\" font-size=\"13\" font-weight=\"bold\">{domain}</text>')\n",
" out.append(f'<text x=\"16\" y=\"42\" fill=\"#555\">{title}</text>')\n",
" pos = {}\n",
" for li, lvl in enumerate(levels):\n",
" y = height - 70 - li * lv\n",
" span = width / (len(lvl) + 1)\n",
" for i, node in enumerate(lvl):\n",
" x = span * (i + 1)\n",
" pos[(li, i)] = (x, y)\n",
" hl = highlight and li == 0 and i == highlight[0]\n",
" sib = highlight and (li, i) in highlight[1]\n",
" fill = \"#fdf0da\" if sib else (\"#e2f2e9\" if hl else \"#f4f4f6\")\n",
" stroke = \"#a86a10\" if sib else (\"#1e7f4f\" if hl else \"#999\")\n",
" out.append(f'<rect x=\"{x-52}\" y=\"{y-16}\" width=\"104\" height=\"32\" fill=\"{fill}\" stroke=\"{stroke}\" stroke-width=\"{2 if (hl or sib) else 1}\" rx=\"4\"/>')\n",
" label = (\"leaf %d\" % i) if li == 0 else (\"root\" if li == len(levels) - 1 else \"node\")\n",
" out.append(f'<text x=\"{x}\" y=\"{y-3}\" text-anchor=\"middle\" fill=\"#333\">{label}</text>')\n",
" out.append(f'<text x=\"{x}\" y=\"{y+11}\" text-anchor=\"middle\" fill=\"#777\">{node.hex()[:10]}\u2026</text>')\n",
" if li > 0:\n",
" for ci in (2 * i, 2 * i + 1):\n",
" if (li - 1, ci) in pos:\n",
" cx, cy = pos[(li - 1, ci)]\n",
" out.append(f'<line x1=\"{x}\" y1=\"{y+16}\" x2=\"{cx}\" y2=\"{cy-16}\" stroke=\"#bbb\"/>')\n",
" rx, ry = pos[(len(levels) - 1, 0)]\n",
" out.append(f'<rect x=\"{rx-135}\" y=\"{ry-62}\" width=\"270\" height=\"30\" fill=\"#eef\" stroke=\"#3b4d8f\" rx=\"4\"/>')\n",
" out.append(f'<text x=\"{rx}\" y=\"{ry-42}\" text-anchor=\"middle\" fill=\"#3b4d8f\">Signed Tree Head: Ed25519(root) via merkleized library</text>')\n",
" out.append(f'<line x1=\"{rx}\" y1=\"{ry-32}\" x2=\"{rx}\" y2=\"{ry-16}\" stroke=\"#3b4d8f\"/>')\n",
" out.append(\"</svg>\")\n",
" return \"\".join(out)\n",
"\n",
"entries = log.entries()\n",
"leaf_hexes = [leaf_hash(e.leaf_bytes()).hex() for e in entries]\n",
"svg = svg_merkle(leaf_hexes, title=f\"your lecture log: {len(entries)} attestation leaves, root {merkle_root([e.leaf_bytes() for e in entries]).hex()[:16]}\u2026\")\n",
"try:\n",
" from IPython.display import SVG, display\n",
" display(SVG(svg))\n",
"except Exception:\n",
" print(svg[:200], \"\u2026 (open in a notebook to render)\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Why a singleton? The design justification\n",
"\n",
"| | Provider (this notebook) | Agent (next notebook) |\n",
"|---|---|---|\n",
"| How many | **exactly one** per log | unbounded |\n",
"| Owns | the signing key, the Lean toolchain, the full log | the provider's PUBLIC key, a pin file |\n",
"| Pays | hours of kernel time per repo, ONCE | milliseconds per check, forever |\n",
"| Obligations | append-only, sign every head, serve proofs, self-verify | pin every head, demand consistency |\n",
"| Can be wrong? | detectably: signatures + pins make lies attributable | fails closed |\n",
"\n",
"The asymmetry is the product. If agents had to run Lean, the service\n",
"would add nothing; if the provider's claims weren't pinned and\n",
"signed, trust would be a rumor. Every artifact in this course lives\n",
"on exactly one side of this table - and the split between this\n",
"notebook and the next MIRRORS it on purpose: if you cannot say\n",
"which notebook a step belongs to, you have not understood the step.\n",
"\n",
"## Exercises\n",
"\n",
"- Append a fifth attestation (edit one field of a copy) and watch the root change; which internal nodes changed and which did not? Explain from the tree shape.\n",
"- The self-inclusion check ran against the tree BEFORE your key existed in any leaf. What does `signing_provenance.self_inclusion` say, and why is recording that honest?\n",
"- Cost accounting: with 4 repos x 30 minutes of Lean and N agents x 5 ms of verification, at what N does the provider model beat every-agent-verifies-locally? (Hint: N=1.)\n",
"- Design question: what breaks if there are TWO providers with one key? With two keys and one log?\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"name": "python",
"pygments_lexer": "ipython3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}

View file

@ -0,0 +1,245 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Lecture 6b: THE AGENT'S SIDE - Verifying Inclusion (one of many)\n",
"\n",
"> **DOMAIN BANNER - read this first.** In this notebook YOU ARE AN\n",
"> AGENT. There are **many** of you. You possess exactly three\n",
"> things: the provider's public key, the evidence files, and about\n",
"> forty lines of hashing code. You do NOT possess Lean, a proof\n",
"> toolchain, or the provider's private key - and you never will\n",
"> need them. Everything below runs in milliseconds. If a cell in\n",
"> this notebook needed Lean, the design would have failed.\n",
"\n",
"This is **Merkle proof verification, not Lean verification** - the\n",
"agent checks WHERE a statement sits, never re-derives WHY it is true.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Learning Objectives\n",
"\n",
"- Implement the complete inclusion verifier from scratch - hashlib only, no pacta imports for the core.\n",
"- Verify a REAL receipt against the REAL signed tree head.\n",
"- See the inclusion path in the picture of the real 8-leaf log.\n",
"- Read the provider's self-check (\"dogfood in both directions\") from the signature block and say what it does and does not prove.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## The whole verifier, from scratch\n",
"\n",
"To make the cost asymmetry unmistakable, here is the ENTIRE core of\n",
"what an agent must implement - RFC 9162 inclusion verification in\n",
"~25 lines of standard-library Python. Read every line; this is all\n",
"the cryptographic machinery your trust rests on (plus one Ed25519\n",
"signature check):\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import hashlib\n",
"\n",
"def leaf_hash(data: bytes) -> bytes:\n",
" return hashlib.sha256(b\"\\x00\" + data).digest()\n",
"\n",
"def node_hash(left: bytes, right: bytes) -> bytes:\n",
" return hashlib.sha256(b\"\\x01\" + left + right).digest()\n",
"\n",
"def verify_inclusion(leaf: bytes, index: int, size: int, proof: list, root: bytes) -> bool:\n",
" if index >= size:\n",
" return False\n",
" fn, sn = index, size - 1\n",
" node = leaf_hash(leaf)\n",
" for sibling in proof:\n",
" if sn == 0:\n",
" return False\n",
" if fn % 2 == 1 or fn == sn:\n",
" node = node_hash(sibling, node)\n",
" if fn % 2 == 0:\n",
" while fn % 2 == 0 and fn != 0:\n",
" fn //= 2\n",
" sn //= 2\n",
" else:\n",
" node = node_hash(node, sibling)\n",
" fn //= 2\n",
" sn //= 2\n",
" return sn == 0 and node == root\n",
"\n",
"print(\"the agent's entire Merkle toolbox: 3 functions,\", \"no imports beyond hashlib\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Apply it to the REAL receipt\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import json\n",
"from pathlib import Path\n",
"import sys\n",
"\n",
"repo_root = Path.cwd()\n",
"if not (repo_root / \"src\" / \"pacta\").exists():\n",
" repo_root = repo_root.parent\n",
"sys.path.insert(0, str(repo_root / \"src\"))\n",
"from pacta.yamlio import load_data\n",
"from pacta.signing import canonical_json\n",
"\n",
"att = load_data(repo_root / \"evidence\" / \"dalek-ed25519.attestation.yaml\")\n",
"receipt = load_data(repo_root / \"evidence\" / \"dalek-ed25519.receipt.yaml\")\n",
"\n",
"leaf_bytes = canonical_json({\"schema_version\": 1, \"type\": \"pacta.transparency.attestation_leaf.v1\", \"attestation\": att})\n",
"proof = [bytes.fromhex(h) for h in receipt[\"inclusion_proof\"]]\n",
"root = bytes.fromhex(receipt[\"sth\"][\"root_hash\"])\n",
"\n",
"ok = verify_inclusion(leaf_bytes, receipt[\"leaf_index\"], receipt[\"tree_size\"], proof, root)\n",
"print(f\"leaf {receipt['leaf_index']} of {receipt['tree_size']}, {len(proof)} siblings -> inclusion:\", ok)\n",
"assert ok\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## One signature check completes the chain\n",
"\n",
"Inclusion binds the attestation to a root; the signature binds the\n",
"root to the provider. Note what the AGENT learns from the signature\n",
"block's `signing_provenance`: the provider signed this root with the\n",
"merkleized library and Merkle-verified that library's own leaf first\n",
"- dogfood in both directions. The agent still re-checks inclusion\n",
"itself (above); the provenance is the provider's discipline made\n",
"visible, not a substitute for the agent's check.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from pacta.transparency import verify_signed_tree_head\n",
"from pacta.sthstore import check_sth_against_store\n",
"import tempfile\n",
"\n",
"ok, diagnostics, statuses = verify_signed_tree_head(receipt[\"sth\"], repo_root / \"evidence\" / \"provider.ed25519.pub\")\n",
"print(\"STH signature:\", statuses.get(\"ed25519\"), \"| verified on backend:\", statuses.get(\"ed25519_backend\"))\n",
"print(\"provider's own discipline, as recorded in the signature block:\")\n",
"print(json.dumps(receipt[\"sth\"][\"signatures\"][\"ed25519\"].get(\"signing_provenance\", {}), indent=1))\n",
"with tempfile.TemporaryDirectory() as tmp:\n",
" pin = check_sth_against_store(receipt[\"sth\"], Path(tmp) / \"pins.json\")\n",
" print(\"pin store:\", pin.action)\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## The picture: where your leaf sits in the real log\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# reconstruct the on-path node hashes from the receipt alone - the\n",
"# agent never needs the other leaves, only the siblings.\n",
"def svg_inclusion(receipt, width=980):\n",
" size, index = receipt[\"tree_size\"], receipt[\"leaf_index\"]\n",
" proof = receipt[\"inclusion_proof\"]\n",
" rows = [f'<svg xmlns=\"http://www.w3.org/2000/svg\" width=\"{width}\" height=\"330\" font-family=\"monospace\" font-size=\"11\">']\n",
" rows.append(f'<rect x=\"4\" y=\"4\" width=\"{width-8}\" height=\"322\" fill=\"none\" stroke=\"#3b4d8f\" stroke-width=\"2\" rx=\"8\"/>')\n",
" rows.append('<text x=\"16\" y=\"24\" fill=\"#3b4d8f\" font-size=\"13\" font-weight=\"bold\">AGENT: verifies only the highlighted path - everything grey is somebody else&apos;s data</text>')\n",
" span = width / (size + 1)\n",
" for i in range(size):\n",
" x = span * (i + 1)\n",
" me = i == index\n",
" fill = \"#e2f2e9\" if me else \"#f4f4f6\"\n",
" stroke = \"#1e7f4f\" if me else \"#bbb\"\n",
" rows.append(f'<rect x=\"{x-42}\" y=\"250\" width=\"84\" height=\"30\" fill=\"{fill}\" stroke=\"{stroke}\" stroke-width=\"{2 if me else 1}\" rx=\"4\"/>')\n",
" label = \"YOUR leaf\" if me else f\"leaf {i}\"\n",
" rows.append(f'<text x=\"{x}\" y=\"269\" text-anchor=\"middle\" fill=\"{stroke}\">{label}</text>')\n",
" node = leaf_hash(leaf_bytes)\n",
" y = 250\n",
" x = span * (index + 1)\n",
" for depth, sib in enumerate(proof):\n",
" ny = y - 60\n",
" nx = x # visual simplification: path rises vertically\n",
" rows.append(f'<rect x=\"{nx-52}\" y=\"{ny-16}\" width=\"104\" height=\"30\" fill=\"#fdf0da\" stroke=\"#a86a10\" rx=\"4\"/>')\n",
" rows.append(f'<text x=\"{nx}\" y=\"{ny+3}\" text-anchor=\"middle\" fill=\"#a86a10\">sibling {depth}: {sib[:10]}\u2026</text>')\n",
" rows.append(f'<line x1=\"{x}\" y1=\"{y-16 if depth else 250}\" x2=\"{nx}\" y2=\"{ny+14}\" stroke=\"#a86a10\"/>')\n",
" y = ny\n",
" rows.append(f'<rect x=\"{x-135}\" y=\"{y-70}\" width=\"270\" height=\"30\" fill=\"#eef\" stroke=\"#3b4d8f\" rx=\"4\"/>')\n",
" rows.append(f'<text x=\"{x}\" y=\"{y-50}\" text-anchor=\"middle\" fill=\"#3b4d8f\">signed root {receipt[\"sth\"][\"root_hash\"][:14]}\u2026 (dogfood-signed)</text>')\n",
" rows.append(f'<line x1=\"{x}\" y1=\"{y-16}\" x2=\"{x}\" y2=\"{y-40}\" stroke=\"#3b4d8f\"/>')\n",
" rows.append(\"</svg>\")\n",
" return \"\".join(rows)\n",
"\n",
"svg = svg_inclusion(receipt)\n",
"try:\n",
" from IPython.display import SVG, display\n",
" display(SVG(svg))\n",
"except Exception:\n",
" print(\"(open in a notebook to render the figure)\")\n",
"print(f\"cost of everything in this notebook: ~{receipt['tree_size'].bit_length()} hashes + 1 signature check - milliseconds.\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Convinced - of what, exactly?\n",
"\n",
"After these cells pass, the agent knows: *the provider whose key I\n",
"pinned states that the Lean proofs of repository X at commit Y check\n",
"out with exactly the documented assumptions, and that statement is\n",
"irrevocably part of the log every other agent sees.* The agent then\n",
"clones commit Y (the git hash IS the content hash) and builds it -\n",
"compiler and build remain declared trusted base until R5. Where a\n",
"claim lives (this notebook) and why it is true (the provider's Lean\n",
"replay, lecture 6a) never blur.\n",
"\n",
"## Exercises\n",
"\n",
"- Flip one byte of the leaf and re-run: which of the ~25 lines catches it?\n",
"- Flip one byte of the ROOT instead: what fails now, inclusion or the signature?\n",
"- Your entire verifier fits in one cell. List everything it does NOT check (freshness? consistency with your previous pin? provider honesty about Lean?) and name the lecture that closes each gap.\n",
"- The provenance block says the provider checked itself. Why must the agent still run its own inclusion check rather than trust that field?\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"name": "python",
"pygments_lexer": "ipython3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}

View file

@ -139,7 +139,16 @@ COURSE = {
5. `05_third_party_attestation_provider.ipynb`
Learn how a proof-checking service can transform hard local verification into provider trust.
6. `06_merkle_transparency_logs.ipynb`
6. `06_merkle_transparency_logs.ipynb`, then the MIRRORED PAIR
`06a_provider_build_the_log.ipynb` / `06b_agent_verify_inclusion.ipynb`
The trust architecture has exactly two domains - ONE provider
who builds and signs the authenticated structure (and pays the
Lean bill), MANY agents who verify inclusion proofs in
milliseconds. The course mirrors that split structurally: 6a is
written entirely in the provider's voice, 6b entirely in the
agent's. If you cannot say which notebook a step belongs to,
you have not understood the step.
Build the Merkle accumulator intuition behind inclusion proofs, consistency proofs, and Signed Tree Heads.
7. `07_agent_consequences.ipynb`
@ -1043,6 +1052,18 @@ COURSE = {
),
md(
"""
## Two domains, two notebooks - by design
Everything above is the shared VOCABULARY. The system itself has
exactly two roles, and the next two notebooks separate them on
purpose: **6a - the provider** (a singleton: builds every leaf via
Lean replay, builds the tree, signs the root with the merkleized
library, and Merkle-verifies its own signing library's leaf before
signing), and **6b - the agent** (one of many: the provider's
public key, the evidence files, ~25 lines of hashing, and nothing
else - explicitly NO Lean). Keep the mirror in mind as you drill
the primitives below; each drill belongs to one side.
## Split Views: why a receipt is not enough
Everything above verifies ONE receipt against ONE signed tree head. A malicious provider can maintain TWO trees - one shown to you, one shown to the world - and both views verify perfectly in isolation. This is EQUIVOCATION, and the defense is memory: pin every tree head you accept, and demand that every later tree head be CONSISTENT with your pin (same size -> same root; larger size -> a verified consistency proof from your pinned size; smaller size -> rollback, reject forever).
@ -1123,6 +1144,421 @@ COURSE = {
),
]
),
"06a_provider_build_the_log.ipynb": notebook(
[
md(
"""
# Lecture 6a: THE PROVIDER'S SIDE - Building the Authenticated Structure
> **DOMAIN BANNER - read this first.** In this notebook YOU ARE THE
> PROVIDER. There is exactly **one** of you per log. You hold the
> signing key. You own a Lean toolchain and hours of compute. You
> carry the append-only obligations. Nothing in this notebook is
> ever executed by an agent - and that asymmetry is not an
> implementation detail, it is the entire design (see the
> justification at the end).
The provider's job, end to end: **verify -> leaf -> tree -> sign ->
self-check**. Only the first step involves Lean; everything after
is hashing and one signature.
"""
),
md(
"""
## Learning Objectives
- Build the full authenticated data structure from real attestations: leaves, tree, Signed Tree Head.
- Place the Lean verification correctly: it is the LEAF-MAKING step, the only expensive one, and it never travels to the agent.
- Sign the root with the merkleized library and run the provider's own inclusion self-check ("the provider eats its own dogfood").
- Justify the singleton/many split as a design decision.
"""
),
md(
"""
## Step 1 - Verify (the expensive step, done ONCE)
The leaf content is a signed **attestation**: the outcome of replaying
every Lean proof of one repository under lean-guard (~30 minutes of
kernel re-checking per fork on the reference machine). This notebook
does NOT re-run that - the shipped `evidence/` attestations ARE that
step's output. What matters architecturally: **the Lean cost lives
here and only here.** No agent ever pays it again.
"""
),
code(
"""
from pathlib import Path
import sys
repo_root = Path.cwd()
if not (repo_root / "src" / "pacta").exists():
repo_root = repo_root.parent
sys.path.insert(0, str(repo_root / "src"))
sys.path.insert(0, str(repo_root / "provider" / "src"))
from pacta.yamlio import load_data
attestations = {
fork: load_data(repo_root / "evidence" / f"{fork}-ed25519.attestation.yaml")
for fork in ["dalek", "anza", "risc0", "betrusted"]
}
for fork, att in attestations.items():
certs = att["certificates"]
clean = sum(1 for c in certs if c["status"] == "proven" and c["axiom_status"] == "clean")
print(f"{fork}: {clean}/{len(certs)} proven | commit {att['subject']['repo_commit'][:8]} | guard: {att['machine_protection']['lean_guard'].rsplit('/',1)[-1]}")
"""
),
md(
"""
## Steps 2+3 - Leaf and tree (cheap, mechanical)
Each attestation is wrapped, canonically serialized, and hashed with
the RFC 9162 leaf prefix `0x00`; pairs of nodes hash with prefix
`0x01`. Build a REAL provider log in a scratch directory - you are
the provider, so mint your own key first:
"""
),
code(
"""
import tempfile
from pacta.signing import generate_ed25519_keypair
from pacta_provider.transparency_log import TransparencyLog
state = Path(tempfile.mkdtemp(prefix="provider-lecture-"))
generate_ed25519_keypair(state / "provider.key", state / "provider.pub")
log = TransparencyLog(state / "log")
log.init("lecture-provider", state / "provider.pub")
receipts = {}
for fork, att in attestations.items():
att_path = state / f"{fork}.attestation.yaml"
from pacta.yamlio import dump_data
dump_data(att, att_path)
receipts[fork] = log.append_attestation(att_path, state / "provider.key", state / "provider.pub")
print("tree size:", receipts["betrusted"]["tree_size"])
print("root:", receipts["betrusted"]["sth"]["root_hash"][:32], "")
"""
),
md(
"""
## Steps 4+5 - Sign the root, then CHECK YOURSELF
The tree head is signed with the **merkleized library itself** (when
the dogfood binary is built): the Ed25519 code that signs this root
is the same pinned dalek source whose proof attestation is a leaf of
this very tree. Before signing, the provider runs the SAME Merkle
inclusion verification an agent would run - on its own signing
library's leaf, against the tree it is about to sign - and embeds
the verdict in the signature block. A root signature that names the
leaf vouching for the code that produced it:
"""
),
code(
"""
import json
sth = receipts["betrusted"]["sth"]
ed = sth["signatures"]["ed25519"]
print("signing backend:", ed.get("signing_backend"))
print(json.dumps(ed.get("signing_provenance", {"note": "dogfood binary not built on this host - OpenSSL fallback, provenance omitted"}), indent=1))
"""
),
md(
"""
## The structure you just built, drawn from your own log
"""
),
code(
"""
from pacta.transparency import leaf_hash, merkle_root, node_hash
def svg_merkle(leaf_hashes, highlight=None, title="", domain="PROVIDER: builds every box below", color="#1e7f4f"):
n = len(leaf_hashes)
width, lh, lv = 980, 108, 92
levels = []
level = [bytes.fromhex(h) if isinstance(h, str) else h for h in leaf_hashes]
levels.append(level)
while len(level) > 1:
nxt = []
for i in range(0, len(level) - 1, 2):
nxt.append(node_hash(level[i], level[i + 1]))
if len(level) % 2:
nxt.append(level[-1])
levels.append(nxt)
level = nxt
height = 130 + lv * len(levels) + 60
out = [f'<svg xmlns="http://www.w3.org/2000/svg" width="{width}" height="{height}" font-family="monospace" font-size="11">']
out.append(f'<rect x="4" y="4" width="{width-8}" height="{height-8}" fill="none" stroke="{color}" stroke-width="2" rx="8"/>')
out.append(f'<text x="16" y="24" fill="{color}" font-size="13" font-weight="bold">{domain}</text>')
out.append(f'<text x="16" y="42" fill="#555">{title}</text>')
pos = {}
for li, lvl in enumerate(levels):
y = height - 70 - li * lv
span = width / (len(lvl) + 1)
for i, node in enumerate(lvl):
x = span * (i + 1)
pos[(li, i)] = (x, y)
hl = highlight and li == 0 and i == highlight[0]
sib = highlight and (li, i) in highlight[1]
fill = "#fdf0da" if sib else ("#e2f2e9" if hl else "#f4f4f6")
stroke = "#a86a10" if sib else ("#1e7f4f" if hl else "#999")
out.append(f'<rect x="{x-52}" y="{y-16}" width="104" height="32" fill="{fill}" stroke="{stroke}" stroke-width="{2 if (hl or sib) else 1}" rx="4"/>')
label = ("leaf %d" % i) if li == 0 else ("root" if li == len(levels) - 1 else "node")
out.append(f'<text x="{x}" y="{y-3}" text-anchor="middle" fill="#333">{label}</text>')
out.append(f'<text x="{x}" y="{y+11}" text-anchor="middle" fill="#777">{node.hex()[:10]}…</text>')
if li > 0:
for ci in (2 * i, 2 * i + 1):
if (li - 1, ci) in pos:
cx, cy = pos[(li - 1, ci)]
out.append(f'<line x1="{x}" y1="{y+16}" x2="{cx}" y2="{cy-16}" stroke="#bbb"/>')
rx, ry = pos[(len(levels) - 1, 0)]
out.append(f'<rect x="{rx-135}" y="{ry-62}" width="270" height="30" fill="#eef" stroke="#3b4d8f" rx="4"/>')
out.append(f'<text x="{rx}" y="{ry-42}" text-anchor="middle" fill="#3b4d8f">Signed Tree Head: Ed25519(root) via merkleized library</text>')
out.append(f'<line x1="{rx}" y1="{ry-32}" x2="{rx}" y2="{ry-16}" stroke="#3b4d8f"/>')
out.append("</svg>")
return "".join(out)
entries = log.entries()
leaf_hexes = [leaf_hash(e.leaf_bytes()).hex() for e in entries]
svg = svg_merkle(leaf_hexes, title=f"your lecture log: {len(entries)} attestation leaves, root {merkle_root([e.leaf_bytes() for e in entries]).hex()[:16]}")
try:
from IPython.display import SVG, display
display(SVG(svg))
except Exception:
print(svg[:200], "… (open in a notebook to render)")
"""
),
md(
"""
## Why a singleton? The design justification
| | Provider (this notebook) | Agent (next notebook) |
|---|---|---|
| How many | **exactly one** per log | unbounded |
| Owns | the signing key, the Lean toolchain, the full log | the provider's PUBLIC key, a pin file |
| Pays | hours of kernel time per repo, ONCE | milliseconds per check, forever |
| Obligations | append-only, sign every head, serve proofs, self-verify | pin every head, demand consistency |
| Can be wrong? | detectably: signatures + pins make lies attributable | fails closed |
The asymmetry is the product. If agents had to run Lean, the service
would add nothing; if the provider's claims weren't pinned and
signed, trust would be a rumor. Every artifact in this course lives
on exactly one side of this table - and the split between this
notebook and the next MIRRORS it on purpose: if you cannot say
which notebook a step belongs to, you have not understood the step.
## Exercises
- Append a fifth attestation (edit one field of a copy) and watch the root change; which internal nodes changed and which did not? Explain from the tree shape.
- The self-inclusion check ran against the tree BEFORE your key existed in any leaf. What does `signing_provenance.self_inclusion` say, and why is recording that honest?
- Cost accounting: with 4 repos x 30 minutes of Lean and N agents x 5 ms of verification, at what N does the provider model beat every-agent-verifies-locally? (Hint: N=1.)
- Design question: what breaks if there are TWO providers with one key? With two keys and one log?
"""
),
]
),
"06b_agent_verify_inclusion.ipynb": notebook(
[
md(
"""
# Lecture 6b: THE AGENT'S SIDE - Verifying Inclusion (one of many)
> **DOMAIN BANNER - read this first.** In this notebook YOU ARE AN
> AGENT. There are **many** of you. You possess exactly three
> things: the provider's public key, the evidence files, and about
> forty lines of hashing code. You do NOT possess Lean, a proof
> toolchain, or the provider's private key - and you never will
> need them. Everything below runs in milliseconds. If a cell in
> this notebook needed Lean, the design would have failed.
This is **Merkle proof verification, not Lean verification** - the
agent checks WHERE a statement sits, never re-derives WHY it is true.
"""
),
md(
"""
## Learning Objectives
- Implement the complete inclusion verifier from scratch - hashlib only, no pacta imports for the core.
- Verify a REAL receipt against the REAL signed tree head.
- See the inclusion path in the picture of the real 8-leaf log.
- Read the provider's self-check ("dogfood in both directions") from the signature block and say what it does and does not prove.
"""
),
md(
"""
## The whole verifier, from scratch
To make the cost asymmetry unmistakable, here is the ENTIRE core of
what an agent must implement - RFC 9162 inclusion verification in
~25 lines of standard-library Python. Read every line; this is all
the cryptographic machinery your trust rests on (plus one Ed25519
signature check):
"""
),
code(
"""
import hashlib
def leaf_hash(data: bytes) -> bytes:
return hashlib.sha256(b"\\x00" + data).digest()
def node_hash(left: bytes, right: bytes) -> bytes:
return hashlib.sha256(b"\\x01" + left + right).digest()
def verify_inclusion(leaf: bytes, index: int, size: int, proof: list, root: bytes) -> bool:
if index >= size:
return False
fn, sn = index, size - 1
node = leaf_hash(leaf)
for sibling in proof:
if sn == 0:
return False
if fn % 2 == 1 or fn == sn:
node = node_hash(sibling, node)
if fn % 2 == 0:
while fn % 2 == 0 and fn != 0:
fn //= 2
sn //= 2
else:
node = node_hash(node, sibling)
fn //= 2
sn //= 2
return sn == 0 and node == root
print("the agent's entire Merkle toolbox: 3 functions,", "no imports beyond hashlib")
"""
),
md(
"""
## Apply it to the REAL receipt
"""
),
code(
"""
import json
from pathlib import Path
import sys
repo_root = Path.cwd()
if not (repo_root / "src" / "pacta").exists():
repo_root = repo_root.parent
sys.path.insert(0, str(repo_root / "src"))
from pacta.yamlio import load_data
from pacta.signing import canonical_json
att = load_data(repo_root / "evidence" / "dalek-ed25519.attestation.yaml")
receipt = load_data(repo_root / "evidence" / "dalek-ed25519.receipt.yaml")
leaf_bytes = canonical_json({"schema_version": 1, "type": "pacta.transparency.attestation_leaf.v1", "attestation": att})
proof = [bytes.fromhex(h) for h in receipt["inclusion_proof"]]
root = bytes.fromhex(receipt["sth"]["root_hash"])
ok = verify_inclusion(leaf_bytes, receipt["leaf_index"], receipt["tree_size"], proof, root)
print(f"leaf {receipt['leaf_index']} of {receipt['tree_size']}, {len(proof)} siblings -> inclusion:", ok)
assert ok
"""
),
md(
"""
## One signature check completes the chain
Inclusion binds the attestation to a root; the signature binds the
root to the provider. Note what the AGENT learns from the signature
block's `signing_provenance`: the provider signed this root with the
merkleized library and Merkle-verified that library's own leaf first
- dogfood in both directions. The agent still re-checks inclusion
itself (above); the provenance is the provider's discipline made
visible, not a substitute for the agent's check.
"""
),
code(
"""
from pacta.transparency import verify_signed_tree_head
from pacta.sthstore import check_sth_against_store
import tempfile
ok, diagnostics, statuses = verify_signed_tree_head(receipt["sth"], repo_root / "evidence" / "provider.ed25519.pub")
print("STH signature:", statuses.get("ed25519"), "| verified on backend:", statuses.get("ed25519_backend"))
print("provider's own discipline, as recorded in the signature block:")
print(json.dumps(receipt["sth"]["signatures"]["ed25519"].get("signing_provenance", {}), indent=1))
with tempfile.TemporaryDirectory() as tmp:
pin = check_sth_against_store(receipt["sth"], Path(tmp) / "pins.json")
print("pin store:", pin.action)
"""
),
md(
"""
## The picture: where your leaf sits in the real log
"""
),
code(
"""
# reconstruct the on-path node hashes from the receipt alone - the
# agent never needs the other leaves, only the siblings.
def svg_inclusion(receipt, width=980):
size, index = receipt["tree_size"], receipt["leaf_index"]
proof = receipt["inclusion_proof"]
rows = [f'<svg xmlns="http://www.w3.org/2000/svg" width="{width}" height="330" font-family="monospace" font-size="11">']
rows.append(f'<rect x="4" y="4" width="{width-8}" height="322" fill="none" stroke="#3b4d8f" stroke-width="2" rx="8"/>')
rows.append('<text x="16" y="24" fill="#3b4d8f" font-size="13" font-weight="bold">AGENT: verifies only the highlighted path - everything grey is somebody else&apos;s data</text>')
span = width / (size + 1)
for i in range(size):
x = span * (i + 1)
me = i == index
fill = "#e2f2e9" if me else "#f4f4f6"
stroke = "#1e7f4f" if me else "#bbb"
rows.append(f'<rect x="{x-42}" y="250" width="84" height="30" fill="{fill}" stroke="{stroke}" stroke-width="{2 if me else 1}" rx="4"/>')
label = "YOUR leaf" if me else f"leaf {i}"
rows.append(f'<text x="{x}" y="269" text-anchor="middle" fill="{stroke}">{label}</text>')
node = leaf_hash(leaf_bytes)
y = 250
x = span * (index + 1)
for depth, sib in enumerate(proof):
ny = y - 60
nx = x # visual simplification: path rises vertically
rows.append(f'<rect x="{nx-52}" y="{ny-16}" width="104" height="30" fill="#fdf0da" stroke="#a86a10" rx="4"/>')
rows.append(f'<text x="{nx}" y="{ny+3}" text-anchor="middle" fill="#a86a10">sibling {depth}: {sib[:10]}…</text>')
rows.append(f'<line x1="{x}" y1="{y-16 if depth else 250}" x2="{nx}" y2="{ny+14}" stroke="#a86a10"/>')
y = ny
rows.append(f'<rect x="{x-135}" y="{y-70}" width="270" height="30" fill="#eef" stroke="#3b4d8f" rx="4"/>')
rows.append(f'<text x="{x}" y="{y-50}" text-anchor="middle" fill="#3b4d8f">signed root {receipt["sth"]["root_hash"][:14]}… (dogfood-signed)</text>')
rows.append(f'<line x1="{x}" y1="{y-16}" x2="{x}" y2="{y-40}" stroke="#3b4d8f"/>')
rows.append("</svg>")
return "".join(rows)
svg = svg_inclusion(receipt)
try:
from IPython.display import SVG, display
display(SVG(svg))
except Exception:
print("(open in a notebook to render the figure)")
print(f"cost of everything in this notebook: ~{receipt['tree_size'].bit_length()} hashes + 1 signature check - milliseconds.")
"""
),
md(
"""
## Convinced - of what, exactly?
After these cells pass, the agent knows: *the provider whose key I
pinned states that the Lean proofs of repository X at commit Y check
out with exactly the documented assumptions, and that statement is
irrevocably part of the log every other agent sees.* The agent then
clones commit Y (the git hash IS the content hash) and builds it -
compiler and build remain declared trusted base until R5. Where a
claim lives (this notebook) and why it is true (the provider's Lean
replay, lecture 6a) never blur.
## Exercises
- Flip one byte of the leaf and re-run: which of the ~25 lines catches it?
- Flip one byte of the ROOT instead: what fails now, inclusion or the signature?
- Your entire verifier fits in one cell. List everything it does NOT check (freshness? consistency with your previous pin? provider honesty about Lean?) and name the lecture that closes each gap.
- The provenance block says the provider checked itself. Why must the agent still run its own inclusion check rather than trust that field?
"""
),
]
),
"07_agent_consequences.ipynb": notebook(
[
md(

View file

@ -10,6 +10,8 @@ EXPECTED_NOTEBOOKS = [
"04_proof_hygiene_and_boundaries.ipynb",
"05_third_party_attestation_provider.ipynb",
"06_merkle_transparency_logs.ipynb",
"06a_provider_build_the_log.ipynb",
"06b_agent_verify_inclusion.ipynb",
"07_agent_consequences.ipynb",
"08_capstone_research_program.ipynb",
"09_dogfood_verified_crypto.ipynb",