proof-aware-crypto-tooling-.../notebooks/06_merkle_transparency_logs.ipynb

279 lines
12 KiB
Text
Raw Normal View History

{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Lecture 6: Merkle Transparency Logs\n",
"\n",
"Transparency logs make signed statements auditable. PACTA uses an RFC 9162-style Merkle accumulator over signed proof-check attestations. A provider signs the tree head, and an agent verifies an inclusion proof before acting on the attestation.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Learning Objectives\n",
"\n",
"- Implement leaf and node hashing with domain separation.\n",
"- Compute a Merkle root.\n",
"- Generate and verify inclusion proofs.\n",
"- Generate and verify consistency proofs.\n",
"- Explain Signed Tree Heads and signature policy.\n",
"- Explain why ML-DSA must fail closed when unavailable.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## RFC 9162 Hash Shape\n",
"\n",
"PACTA follows the Certificate Transparency hash structure:\n",
"\n",
"- Empty tree hash: `SHA256(\"\")`\n",
"- Leaf hash: `SHA256(0x00 || leaf_input)`\n",
"- Node hash: `SHA256(0x01 || left || right)`\n",
"\n",
"The prefix bytes prevent a leaf value from being confused with an internal node value.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import sys\n",
"\n",
"repo_root = Path.cwd()\n",
"if not (repo_root / \"src\" / \"pacta\").exists():\n",
" repo_root = repo_root.parent\n",
"sys.path.insert(0, str(repo_root / \"src\"))\n",
"\n",
"from pacta.transparency import (\n",
" leaf_hash,\n",
" node_hash,\n",
" merkle_root,\n",
" inclusion_proof,\n",
" verify_inclusion,\n",
" consistency_proof,\n",
" verify_consistency,\n",
")\n",
"\n",
"leaves = [f\"attestation-{i}\".encode() for i in range(1, 6)]\n",
"root = merkle_root(leaves)\n",
"print(root.hex())\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"for index, leaf in enumerate(leaves):\n",
" proof = inclusion_proof(leaves, index)\n",
" ok = verify_inclusion(leaf, index, len(leaves), proof, root)\n",
" print(index, ok, [node.hex()[:12] for node in proof])\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Consistency Proofs\n",
"\n",
"An inclusion proof answers: \"Is this leaf in this tree?\"\n",
"\n",
"A consistency proof answers: \"Is the newer tree an append-only extension of the older tree?\"\n",
"\n",
"Both are needed for a monitored transparency system. Inclusion is enough for one agent to bind one attestation to one signed tree head. Consistency lets monitors detect equivocation or tree rewrites across time.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"old_size = 3\n",
"old_root = merkle_root(leaves[:old_size])\n",
"new_root = merkle_root(leaves)\n",
"proof = consistency_proof(leaves, old_size)\n",
"print(\"old:\", old_root.hex())\n",
"print(\"new:\", new_root.hex())\n",
"print(\"proof:\", [node.hex()[:12] for node in proof])\n",
"print(\"consistent:\", verify_consistency(old_size, len(leaves), old_root, new_root, proof))\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Signed Tree Heads\n",
"\n",
"A Signed Tree Head records:\n",
"\n",
"- log ID,\n",
"- tree size,\n",
"- timestamp,\n",
"- root hash,\n",
"- hash algorithm,\n",
"- signatures.\n",
"\n",
"PACTA signs the canonical JSON STH payload with Ed25519 through OpenSSL. It also records an ML-DSA-65 slot. On this host, if no real ML-DSA backend is present, the slot is `unavailable`.\n",
"\n",
"Policy matters:\n",
"\n",
"- `require-signatures ed25519`: verify Ed25519 and allow ML-DSA to be unavailable.\n",
"- `require-signatures both`: require Ed25519 and ML-DSA verified. If ML-DSA is unavailable, fail closed.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from pacta.postquantum import detect_ml_dsa\n",
"\n",
"capability = detect_ml_dsa()\n",
"print(capability.available)\n",
"print(capability.backend)\n",
"print(capability.reason)\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Why Ed25519 and ML-DSA Together?\n",
"\n",
"Ed25519 is useful because it is widely deployed, fast, and directly relevant to the Ed25519 proof corpus. That creates a deliberate \"eat your own dogfood\" loop: the proof-checking ecosystem signs evidence using a primitive whose implementation family is under formal scrutiny.\n",
"\n",
"ML-DSA adds post-quantum robustness for the accumulator signature layer. But it must be a real signature, not an aspirational label. If a host lacks ML-DSA, the correct result is an explicit blocker.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
Mirrored lectures 6a/6b: the authenticated structure, drawn and domain-separated The trust architecture has exactly two roles and the curriculum now mirrors that split structurally - the conceptual burden is the design, stated as such to the student: - 06a THE PROVIDER'S SIDE (singleton). Domain banner in the provider's voice. The full build pipeline run live in a scratch log made from the REAL attestations: verify (Lean replay = the leaf-making step, the only expensive one - the shipped evidence IS its output) -> leaf (0x00 domain separation) -> tree -> STH signed via the MERKLEIZED LIBRARY -> the self-inclusion check embedded in the signature block. A generated SVG draws the student's own tree: leaves, internal nodes, root, and the signature box, framed in the provider's domain color. Closes with the singleton-vs-many justification table (key/cost/obligation asymmetry) and exercises. - 06b THE AGENT'S SIDE (one of many). Domain banner in the agent's voice: you own the public key, the evidence files, ~25 lines of hashing - and explicitly NO Lean. The COMPLETE RFC 9162 inclusion verifier is implemented from scratch in one cell (hashlib only, no pacta imports for the core) and run against the REAL dalek receipt (leaf 4 of 8, three siblings, dogfood-signed root); then the STH signature, the provider's signing_provenance read and interpreted (why the agent still re-checks inclusion itself), the pin store, and an SVG of the real log with the agent's path highlighted against the grey leaves it never needs. Cost line: ~4 hashes + 1 signature. - Lecture 06 now routes students into the pair and states the mirror rule ("if you cannot say which notebook a step belongs to, you have not understood the step"); lecture 09 records that dogfood now runs in BOTH directions; course map + README updated. Every cell of 06a/06b/09 executed against the real evidence before commit (SVGs render in Jupyter, fail soft in plain exec). One generation bug found and fixed: a single-backslash \\x00 in the generator produced a literal NUL byte in a cell. 50/50 tests green with the notebook inventory at twelve. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 13:26:32 +00:00
"## Two domains, two notebooks - by design\n",
"\n",
"Everything above is the shared VOCABULARY. The system itself has\n",
"exactly two roles, and the next two notebooks separate them on\n",
"purpose: **6a - the provider** (a singleton: builds every leaf via\n",
"Lean replay, builds the tree, signs the root with the merkleized\n",
"library, and Merkle-verifies its own signing library's leaf before\n",
"signing), and **6b - the agent** (one of many: the provider's\n",
"public key, the evidence files, ~25 lines of hashing, and nothing\n",
"else - explicitly NO Lean). Keep the mirror in mind as you drill\n",
"the primitives below; each drill belongs to one side.\n",
"\n",
Curriculum: the ratchet rule, the four-tier reality, and lecture 9 (dogfood) The notebooks now carry the same didactic contract as the companion book (the "ratchet rule", stated in the course map): every load-bearing idea runs twice - napkin scale, then real scale - and every pair is EXECUTABLE in the notebook, not narrated. - Lecture 1: the truth boundary updated to the proven four-tier apex, with the what-is-still-NOT-proven list (SHA-512, parsers, signing, wallets) given equal weight. - Lecture 2: napkin/real scoring pair - a two-certificate toy card scored in your head, then the shipped sixteen-certificate R4 fixture through the same function, residual blockers and per-tier boundary axioms printed. - Lecture 6: new split-view section. A runnable equivocation drill: pin a two-leaf view, grow it honestly with a consistency proof, then present a forged same-size root and watch the pin store name the attack. Real-scale pointers to --sth-store, log-consistency, log-audit, and the freshness policy; a new exercise asks students to construct the lie a size-only anchor check would miss. - Lecture 7: the wallet gate now swings BOTH ways on real evidence - a partial card denied at R3, the shipped R4 card allowed - both runnable. - Lecture 8 capstone: "design R4" became "audit R4": read the shipped card like an auditor, then design the R5 discharge plan (parser specs, verified SHA-512, signing-side, per-fork production-path mapping). - NEW Lecture 9, "Eat Your Own Dogfood": the honest coverage ledger of the proven-path verifier; a napkin PEM decode (the fixed 12-byte Ed25519 SPKI prefix, read with your eyes) paired with the mechanical extraction; live backend dispatch; the fail-closed --require-verified-verifier policy; and the hybrid-PQC section - proven-classical Ed25519 plus a required-but-honest ML-DSA slot ("blockers get fixed; placeholders get trusted"). Every code cell of the changed notebooks was executed end-to-end before committing (outputs stripped per house rules). 49/49 tests green with the notebook inventory updated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 08:18:06 +00:00
"## Split Views: why a receipt is not enough\n",
"\n",
"Everything above verifies ONE receipt against ONE signed tree head. A malicious provider can maintain TWO trees - one shown to you, one shown to the world - and both views verify perfectly in isolation. This is EQUIVOCATION, and the defense is memory: pin every tree head you accept, and demand that every later tree head be CONSISTENT with your pin (same size -> same root; larger size -> a verified consistency proof from your pinned size; smaller size -> rollback, reject forever).\n",
"\n",
"pacta implements this as a local STH pin store. Run the whole attack and its detection, napkin-size:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# NAPKIN: pin a 2-leaf view, then let the log grow honestly - and then\n",
"# let a SPLIT VIEW present a different root at the pinned size.\n",
"import tempfile\n",
"from pathlib import Path as _P\n",
"from pacta.sthstore import check_sth_against_store\n",
"from pacta.transparency import consistency_proof, merkle_root, proof_to_hex\n",
"\n",
"honest = [b\"attestation-A\", b\"attestation-B\", b\"attestation-C\"]\n",
"evil = [b\"attestation-A\", b\"attestation-EVIL\", b\"attestation-C\"]\n",
"\n",
"with tempfile.TemporaryDirectory() as tmp:\n",
" store = _P(tmp) / \"sth-store.json\"\n",
" sth = lambda size, leaves: {\n",
" \"log_id\": \"demo-log\", \"tree_size\": size,\n",
" \"root_hash\": merkle_root(leaves[:size]).hex(),\n",
" \"timestamp\": \"2026-07-06T00:00:00Z\",\n",
" }\n",
" print(\"pin: \", check_sth_against_store(sth(2, honest), store).diagnostics[0])\n",
" grown = check_sth_against_store(\n",
" sth(3, honest), store,\n",
" consistency_proof_hex=proof_to_hex(consistency_proof(honest, 2)),\n",
" )\n",
" print(\"grow: \", grown.diagnostics[0])\n",
" attack = check_sth_against_store(sth(3, evil), store)\n",
" print(\"attack ok?\", attack.ok)\n",
" print(\"verdict: \", attack.diagnostics[0][:120], \"...\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"At real scale the same check runs on every `pacta receipt-verify --sth-store ...` and `pacta agent --sth-store ...` invocation; receipts embed a consistency anchor from the previous tree size, the provider serves proofs from arbitrary pinned sizes (`pacta_provider log-consistency --from-size N`), and `pacta_provider log-audit` is the monitor's self-check. A freshness policy (`--max-sth-age-seconds`) closes the stale-root hole: an old-but-valid tree head could hide later entries.\n",
"\n",
REAL EVIDENCE: guarded replay of all four repos, attested, logged, dogfooded The provider ran its full honest replay against the four verified repositories on this machine - every Lean compile and axiom audit routed through lean-guard (memory-capped, core-pinned, single-flight, ~30 min per fork) - and the results are now shipped under evidence/: - 16/16 certificates proven per fork, every axiom cone boundary-exact (the four apex tiers carry their fork's documented SHA-512/wire boundary axiom-for-axiom), each attestation pinned to the exact repo commit (dalek 8ded7bc, anza 673c15e, risc0 98a13a6, betrusted 81f614a) and Ed25519-signed. - All four appended to the persistent transparency log. The log holds EIGHT leaves: the first four are the initial run's attestations, which honestly recorded an AUDIT FAILURE (the two pacta bugs fixed in e87f0e8) - an append-only trust ledger keeps its bad day, and the fixed run's leaves sit beside it. - Every receipt re-verified through the FULL stack: dogfood verifier (backend verified-dalek-serial recorded), STH pin store, freshness policy. Receipts are freshly issued against the final tree (a stale mid-run receipt tripped the pin store's rollback defense exactly as designed; the rollback diagnostic now hints at idempotent re-issue). - The capstone consequence ran for real: pacta agent with trusted provider + signature via the proven path + required receipt + pin store + --require-verified-verifier built the R4-gated library capsule from ATTESTED evidence (no local Lean replay needed by the consuming agent). Docs and teaching updated against the real artifacts: evidence/README (inventory + re-verify instructions), README "Real Evidence" section, lecture 5 now re-derives 16/16 verdicts from the REAL dalek attestation (signature checked on the proven path, provider labels ignored), and lecture 6 verifies all four REAL receipts and walks a fresh pin store over them. Every changed notebook cell executed before commit. 49/49 tests green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 12:54:48 +00:00
"### The real thing\n",
"\n",
"The `evidence/` directory holds four REAL receipts from the shipped transparency log (tree size 8 - the first four leaves honestly record a failed audit run; read `evidence/README.md`). Verify all four cryptographically and watch a fresh pin store handle them:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import tempfile\n",
"from pathlib import Path as _P\n",
"from pacta.sthstore import check_sth_against_store\n",
"from pacta.transparency import verify_receipt\n",
"from pacta.yamlio import load_data as _load\n",
"\n",
"log_key = repo_root / \"evidence\" / \"provider.ed25519.pub\"\n",
"with tempfile.TemporaryDirectory() as tmp:\n",
" store = _P(tmp) / \"pins.json\"\n",
" for fork in [\"dalek\", \"anza\", \"risc0\", \"betrusted\"]:\n",
" att = _load(repo_root / \"evidence\" / f\"{fork}-ed25519.attestation.yaml\")\n",
" receipt = _load(repo_root / \"evidence\" / f\"{fork}-ed25519.receipt.yaml\")\n",
" result = verify_receipt(att, receipt, log_key)\n",
" pin = check_sth_against_store(receipt[\"sth\"], store, consistency_from=receipt.get(\"consistency\"))\n",
" print(f\"{fork}: receipt accepted={result.accepted} backend={result.signatures.get('ed25519_backend')} pin={pin.action}\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Exercises\n",
"\n",
"- Tamper with one leaf and show that inclusion verification fails.\n",
"- Explain why the tree head signature must cover tree size as well as root hash.\n",
Curriculum: the ratchet rule, the four-tier reality, and lecture 9 (dogfood) The notebooks now carry the same didactic contract as the companion book (the "ratchet rule", stated in the course map): every load-bearing idea runs twice - napkin scale, then real scale - and every pair is EXECUTABLE in the notebook, not narrated. - Lecture 1: the truth boundary updated to the proven four-tier apex, with the what-is-still-NOT-proven list (SHA-512, parsers, signing, wallets) given equal weight. - Lecture 2: napkin/real scoring pair - a two-certificate toy card scored in your head, then the shipped sixteen-certificate R4 fixture through the same function, residual blockers and per-tier boundary axioms printed. - Lecture 6: new split-view section. A runnable equivocation drill: pin a two-leaf view, grow it honestly with a consistency proof, then present a forged same-size root and watch the pin store name the attack. Real-scale pointers to --sth-store, log-consistency, log-audit, and the freshness policy; a new exercise asks students to construct the lie a size-only anchor check would miss. - Lecture 7: the wallet gate now swings BOTH ways on real evidence - a partial card denied at R3, the shipped R4 card allowed - both runnable. - Lecture 8 capstone: "design R4" became "audit R4": read the shipped card like an auditor, then design the R5 discharge plan (parser specs, verified SHA-512, signing-side, per-fork production-path mapping). - NEW Lecture 9, "Eat Your Own Dogfood": the honest coverage ledger of the proven-path verifier; a napkin PEM decode (the fixed 12-byte Ed25519 SPKI prefix, read with your eyes) paired with the mechanical extraction; live backend dispatch; the fail-closed --require-verified-verifier policy; and the hybrid-PQC section - proven-classical Ed25519 plus a required-but-honest ML-DSA slot ("blockers get fixed; placeholders get trusted"). Every code cell of the changed notebooks was executed end-to-end before committing (outputs stripped per house rules). 49/49 tests green with the notebook inventory updated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 08:18:06 +00:00
"- Napkin, then real: run the split-view drill above; then initialize a real provider log (`pacta_provider log-init`), append two attestations, and verify the second receipt with `--sth-store` - watch the pin advance with a verified consistency proof.\n",
"- Why must the consistency anchor's ROOT (not just its size) be checked against the pin? Construct the lie that a size-only check would miss.\n",
"- Write a policy for when an autonomous agent should require `both` signatures.\n",
Curriculum: the ratchet rule, the four-tier reality, and lecture 9 (dogfood) The notebooks now carry the same didactic contract as the companion book (the "ratchet rule", stated in the course map): every load-bearing idea runs twice - napkin scale, then real scale - and every pair is EXECUTABLE in the notebook, not narrated. - Lecture 1: the truth boundary updated to the proven four-tier apex, with the what-is-still-NOT-proven list (SHA-512, parsers, signing, wallets) given equal weight. - Lecture 2: napkin/real scoring pair - a two-certificate toy card scored in your head, then the shipped sixteen-certificate R4 fixture through the same function, residual blockers and per-tier boundary axioms printed. - Lecture 6: new split-view section. A runnable equivocation drill: pin a two-leaf view, grow it honestly with a consistency proof, then present a forged same-size root and watch the pin store name the attack. Real-scale pointers to --sth-store, log-consistency, log-audit, and the freshness policy; a new exercise asks students to construct the lie a size-only anchor check would miss. - Lecture 7: the wallet gate now swings BOTH ways on real evidence - a partial card denied at R3, the shipped R4 card allowed - both runnable. - Lecture 8 capstone: "design R4" became "audit R4": read the shipped card like an auditor, then design the R5 discharge plan (parser specs, verified SHA-512, signing-side, per-fork production-path mapping). - NEW Lecture 9, "Eat Your Own Dogfood": the honest coverage ledger of the proven-path verifier; a napkin PEM decode (the fixed 12-byte Ed25519 SPKI prefix, read with your eyes) paired with the mechanical extraction; live backend dispatch; the fail-closed --require-verified-verifier policy; and the hybrid-PQC section - proven-classical Ed25519 plus a required-but-honest ML-DSA slot ("blockers get fixed; placeholders get trusted"). Every code cell of the changed notebooks was executed end-to-end before committing (outputs stripped per house rules). 49/49 tests green with the notebook inventory updated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 08:18:06 +00:00
"- Research checkpoint: compare PACTA's pin store to production Certificate Transparency monitor/gossip requirements - what does gossip add that a single pin store cannot?\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"name": "python",
"pygments_lexer": "ipython3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}