proof-aware-crypto-tooling-.../notebooks/06_merkle_transparency_logs.ipynb

267 lines
11 KiB
Text
Raw Normal View History

{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Lecture 6: Merkle Transparency Logs\n",
"\n",
"Transparency logs make signed statements auditable. PACTA uses an RFC 9162-style Merkle accumulator over signed proof-check attestations. A provider signs the tree head, and an agent verifies an inclusion proof before acting on the attestation.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Learning Objectives\n",
"\n",
"- Implement leaf and node hashing with domain separation.\n",
"- Compute a Merkle root.\n",
"- Generate and verify inclusion proofs.\n",
"- Generate and verify consistency proofs.\n",
"- Explain Signed Tree Heads and signature policy.\n",
"- Explain why ML-DSA must fail closed when unavailable.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## RFC 9162 Hash Shape\n",
"\n",
"PACTA follows the Certificate Transparency hash structure:\n",
"\n",
"- Empty tree hash: `SHA256(\"\")`\n",
"- Leaf hash: `SHA256(0x00 || leaf_input)`\n",
"- Node hash: `SHA256(0x01 || left || right)`\n",
"\n",
"The prefix bytes prevent a leaf value from being confused with an internal node value.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import sys\n",
"\n",
"repo_root = Path.cwd()\n",
"if not (repo_root / \"src\" / \"pacta\").exists():\n",
" repo_root = repo_root.parent\n",
"sys.path.insert(0, str(repo_root / \"src\"))\n",
"\n",
"from pacta.transparency import (\n",
" leaf_hash,\n",
" node_hash,\n",
" merkle_root,\n",
" inclusion_proof,\n",
" verify_inclusion,\n",
" consistency_proof,\n",
" verify_consistency,\n",
")\n",
"\n",
"leaves = [f\"attestation-{i}\".encode() for i in range(1, 6)]\n",
"root = merkle_root(leaves)\n",
"print(root.hex())\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"for index, leaf in enumerate(leaves):\n",
" proof = inclusion_proof(leaves, index)\n",
" ok = verify_inclusion(leaf, index, len(leaves), proof, root)\n",
" print(index, ok, [node.hex()[:12] for node in proof])\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Consistency Proofs\n",
"\n",
"An inclusion proof answers: \"Is this leaf in this tree?\"\n",
"\n",
"A consistency proof answers: \"Is the newer tree an append-only extension of the older tree?\"\n",
"\n",
"Both are needed for a monitored transparency system. Inclusion is enough for one agent to bind one attestation to one signed tree head. Consistency lets monitors detect equivocation or tree rewrites across time.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"old_size = 3\n",
"old_root = merkle_root(leaves[:old_size])\n",
"new_root = merkle_root(leaves)\n",
"proof = consistency_proof(leaves, old_size)\n",
"print(\"old:\", old_root.hex())\n",
"print(\"new:\", new_root.hex())\n",
"print(\"proof:\", [node.hex()[:12] for node in proof])\n",
"print(\"consistent:\", verify_consistency(old_size, len(leaves), old_root, new_root, proof))\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Signed Tree Heads\n",
"\n",
"A Signed Tree Head records:\n",
"\n",
"- log ID,\n",
"- tree size,\n",
"- timestamp,\n",
"- root hash,\n",
"- hash algorithm,\n",
"- signatures.\n",
"\n",
"PACTA signs the canonical JSON STH payload with Ed25519 through OpenSSL. It also records an ML-DSA-65 slot. On this host, if no real ML-DSA backend is present, the slot is `unavailable`.\n",
"\n",
"Policy matters:\n",
"\n",
"- `require-signatures ed25519`: verify Ed25519 and allow ML-DSA to be unavailable.\n",
"- `require-signatures both`: require Ed25519 and ML-DSA verified. If ML-DSA is unavailable, fail closed.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from pacta.postquantum import detect_ml_dsa\n",
"\n",
"capability = detect_ml_dsa()\n",
"print(capability.available)\n",
"print(capability.backend)\n",
"print(capability.reason)\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Why Ed25519 and ML-DSA Together?\n",
"\n",
"Ed25519 is useful because it is widely deployed, fast, and directly relevant to the Ed25519 proof corpus. That creates a deliberate \"eat your own dogfood\" loop: the proof-checking ecosystem signs evidence using a primitive whose implementation family is under formal scrutiny.\n",
"\n",
"ML-DSA adds post-quantum robustness for the accumulator signature layer. But it must be a real signature, not an aspirational label. If a host lacks ML-DSA, the correct result is an explicit blocker.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
Curriculum: the ratchet rule, the four-tier reality, and lecture 9 (dogfood) The notebooks now carry the same didactic contract as the companion book (the "ratchet rule", stated in the course map): every load-bearing idea runs twice - napkin scale, then real scale - and every pair is EXECUTABLE in the notebook, not narrated. - Lecture 1: the truth boundary updated to the proven four-tier apex, with the what-is-still-NOT-proven list (SHA-512, parsers, signing, wallets) given equal weight. - Lecture 2: napkin/real scoring pair - a two-certificate toy card scored in your head, then the shipped sixteen-certificate R4 fixture through the same function, residual blockers and per-tier boundary axioms printed. - Lecture 6: new split-view section. A runnable equivocation drill: pin a two-leaf view, grow it honestly with a consistency proof, then present a forged same-size root and watch the pin store name the attack. Real-scale pointers to --sth-store, log-consistency, log-audit, and the freshness policy; a new exercise asks students to construct the lie a size-only anchor check would miss. - Lecture 7: the wallet gate now swings BOTH ways on real evidence - a partial card denied at R3, the shipped R4 card allowed - both runnable. - Lecture 8 capstone: "design R4" became "audit R4": read the shipped card like an auditor, then design the R5 discharge plan (parser specs, verified SHA-512, signing-side, per-fork production-path mapping). - NEW Lecture 9, "Eat Your Own Dogfood": the honest coverage ledger of the proven-path verifier; a napkin PEM decode (the fixed 12-byte Ed25519 SPKI prefix, read with your eyes) paired with the mechanical extraction; live backend dispatch; the fail-closed --require-verified-verifier policy; and the hybrid-PQC section - proven-classical Ed25519 plus a required-but-honest ML-DSA slot ("blockers get fixed; placeholders get trusted"). Every code cell of the changed notebooks was executed end-to-end before committing (outputs stripped per house rules). 49/49 tests green with the notebook inventory updated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 08:18:06 +00:00
"## Split Views: why a receipt is not enough\n",
"\n",
"Everything above verifies ONE receipt against ONE signed tree head. A malicious provider can maintain TWO trees - one shown to you, one shown to the world - and both views verify perfectly in isolation. This is EQUIVOCATION, and the defense is memory: pin every tree head you accept, and demand that every later tree head be CONSISTENT with your pin (same size -> same root; larger size -> a verified consistency proof from your pinned size; smaller size -> rollback, reject forever).\n",
"\n",
"pacta implements this as a local STH pin store. Run the whole attack and its detection, napkin-size:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# NAPKIN: pin a 2-leaf view, then let the log grow honestly - and then\n",
"# let a SPLIT VIEW present a different root at the pinned size.\n",
"import tempfile\n",
"from pathlib import Path as _P\n",
"from pacta.sthstore import check_sth_against_store\n",
"from pacta.transparency import consistency_proof, merkle_root, proof_to_hex\n",
"\n",
"honest = [b\"attestation-A\", b\"attestation-B\", b\"attestation-C\"]\n",
"evil = [b\"attestation-A\", b\"attestation-EVIL\", b\"attestation-C\"]\n",
"\n",
"with tempfile.TemporaryDirectory() as tmp:\n",
" store = _P(tmp) / \"sth-store.json\"\n",
" sth = lambda size, leaves: {\n",
" \"log_id\": \"demo-log\", \"tree_size\": size,\n",
" \"root_hash\": merkle_root(leaves[:size]).hex(),\n",
" \"timestamp\": \"2026-07-06T00:00:00Z\",\n",
" }\n",
" print(\"pin: \", check_sth_against_store(sth(2, honest), store).diagnostics[0])\n",
" grown = check_sth_against_store(\n",
" sth(3, honest), store,\n",
" consistency_proof_hex=proof_to_hex(consistency_proof(honest, 2)),\n",
" )\n",
" print(\"grow: \", grown.diagnostics[0])\n",
" attack = check_sth_against_store(sth(3, evil), store)\n",
" print(\"attack ok?\", attack.ok)\n",
" print(\"verdict: \", attack.diagnostics[0][:120], \"...\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"At real scale the same check runs on every `pacta receipt-verify --sth-store ...` and `pacta agent --sth-store ...` invocation; receipts embed a consistency anchor from the previous tree size, the provider serves proofs from arbitrary pinned sizes (`pacta_provider log-consistency --from-size N`), and `pacta_provider log-audit` is the monitor's self-check. A freshness policy (`--max-sth-age-seconds`) closes the stale-root hole: an old-but-valid tree head could hide later entries.\n",
"\n",
REAL EVIDENCE: guarded replay of all four repos, attested, logged, dogfooded The provider ran its full honest replay against the four verified repositories on this machine - every Lean compile and axiom audit routed through lean-guard (memory-capped, core-pinned, single-flight, ~30 min per fork) - and the results are now shipped under evidence/: - 16/16 certificates proven per fork, every axiom cone boundary-exact (the four apex tiers carry their fork's documented SHA-512/wire boundary axiom-for-axiom), each attestation pinned to the exact repo commit (dalek 8ded7bc, anza 673c15e, risc0 98a13a6, betrusted 81f614a) and Ed25519-signed. - All four appended to the persistent transparency log. The log holds EIGHT leaves: the first four are the initial run's attestations, which honestly recorded an AUDIT FAILURE (the two pacta bugs fixed in e87f0e8) - an append-only trust ledger keeps its bad day, and the fixed run's leaves sit beside it. - Every receipt re-verified through the FULL stack: dogfood verifier (backend verified-dalek-serial recorded), STH pin store, freshness policy. Receipts are freshly issued against the final tree (a stale mid-run receipt tripped the pin store's rollback defense exactly as designed; the rollback diagnostic now hints at idempotent re-issue). - The capstone consequence ran for real: pacta agent with trusted provider + signature via the proven path + required receipt + pin store + --require-verified-verifier built the R4-gated library capsule from ATTESTED evidence (no local Lean replay needed by the consuming agent). Docs and teaching updated against the real artifacts: evidence/README (inventory + re-verify instructions), README "Real Evidence" section, lecture 5 now re-derives 16/16 verdicts from the REAL dalek attestation (signature checked on the proven path, provider labels ignored), and lecture 6 verifies all four REAL receipts and walks a fresh pin store over them. Every changed notebook cell executed before commit. 49/49 tests green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 12:54:48 +00:00
"### The real thing\n",
"\n",
"The `evidence/` directory holds four REAL receipts from the shipped transparency log (tree size 8 - the first four leaves honestly record a failed audit run; read `evidence/README.md`). Verify all four cryptographically and watch a fresh pin store handle them:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import tempfile\n",
"from pathlib import Path as _P\n",
"from pacta.sthstore import check_sth_against_store\n",
"from pacta.transparency import verify_receipt\n",
"from pacta.yamlio import load_data as _load\n",
"\n",
"log_key = repo_root / \"evidence\" / \"provider.ed25519.pub\"\n",
"with tempfile.TemporaryDirectory() as tmp:\n",
" store = _P(tmp) / \"pins.json\"\n",
" for fork in [\"dalek\", \"anza\", \"risc0\", \"betrusted\"]:\n",
" att = _load(repo_root / \"evidence\" / f\"{fork}-ed25519.attestation.yaml\")\n",
" receipt = _load(repo_root / \"evidence\" / f\"{fork}-ed25519.receipt.yaml\")\n",
" result = verify_receipt(att, receipt, log_key)\n",
" pin = check_sth_against_store(receipt[\"sth\"], store, consistency_from=receipt.get(\"consistency\"))\n",
" print(f\"{fork}: receipt accepted={result.accepted} backend={result.signatures.get('ed25519_backend')} pin={pin.action}\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Exercises\n",
"\n",
"- Tamper with one leaf and show that inclusion verification fails.\n",
"- Explain why the tree head signature must cover tree size as well as root hash.\n",
Curriculum: the ratchet rule, the four-tier reality, and lecture 9 (dogfood) The notebooks now carry the same didactic contract as the companion book (the "ratchet rule", stated in the course map): every load-bearing idea runs twice - napkin scale, then real scale - and every pair is EXECUTABLE in the notebook, not narrated. - Lecture 1: the truth boundary updated to the proven four-tier apex, with the what-is-still-NOT-proven list (SHA-512, parsers, signing, wallets) given equal weight. - Lecture 2: napkin/real scoring pair - a two-certificate toy card scored in your head, then the shipped sixteen-certificate R4 fixture through the same function, residual blockers and per-tier boundary axioms printed. - Lecture 6: new split-view section. A runnable equivocation drill: pin a two-leaf view, grow it honestly with a consistency proof, then present a forged same-size root and watch the pin store name the attack. Real-scale pointers to --sth-store, log-consistency, log-audit, and the freshness policy; a new exercise asks students to construct the lie a size-only anchor check would miss. - Lecture 7: the wallet gate now swings BOTH ways on real evidence - a partial card denied at R3, the shipped R4 card allowed - both runnable. - Lecture 8 capstone: "design R4" became "audit R4": read the shipped card like an auditor, then design the R5 discharge plan (parser specs, verified SHA-512, signing-side, per-fork production-path mapping). - NEW Lecture 9, "Eat Your Own Dogfood": the honest coverage ledger of the proven-path verifier; a napkin PEM decode (the fixed 12-byte Ed25519 SPKI prefix, read with your eyes) paired with the mechanical extraction; live backend dispatch; the fail-closed --require-verified-verifier policy; and the hybrid-PQC section - proven-classical Ed25519 plus a required-but-honest ML-DSA slot ("blockers get fixed; placeholders get trusted"). Every code cell of the changed notebooks was executed end-to-end before committing (outputs stripped per house rules). 49/49 tests green with the notebook inventory updated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 08:18:06 +00:00
"- Napkin, then real: run the split-view drill above; then initialize a real provider log (`pacta_provider log-init`), append two attestations, and verify the second receipt with `--sth-store` - watch the pin advance with a verified consistency proof.\n",
"- Why must the consistency anchor's ROOT (not just its size) be checked against the pin? Construct the lie that a size-only check would miss.\n",
"- Write a policy for when an autonomous agent should require `both` signatures.\n",
Curriculum: the ratchet rule, the four-tier reality, and lecture 9 (dogfood) The notebooks now carry the same didactic contract as the companion book (the "ratchet rule", stated in the course map): every load-bearing idea runs twice - napkin scale, then real scale - and every pair is EXECUTABLE in the notebook, not narrated. - Lecture 1: the truth boundary updated to the proven four-tier apex, with the what-is-still-NOT-proven list (SHA-512, parsers, signing, wallets) given equal weight. - Lecture 2: napkin/real scoring pair - a two-certificate toy card scored in your head, then the shipped sixteen-certificate R4 fixture through the same function, residual blockers and per-tier boundary axioms printed. - Lecture 6: new split-view section. A runnable equivocation drill: pin a two-leaf view, grow it honestly with a consistency proof, then present a forged same-size root and watch the pin store name the attack. Real-scale pointers to --sth-store, log-consistency, log-audit, and the freshness policy; a new exercise asks students to construct the lie a size-only anchor check would miss. - Lecture 7: the wallet gate now swings BOTH ways on real evidence - a partial card denied at R3, the shipped R4 card allowed - both runnable. - Lecture 8 capstone: "design R4" became "audit R4": read the shipped card like an auditor, then design the R5 discharge plan (parser specs, verified SHA-512, signing-side, per-fork production-path mapping). - NEW Lecture 9, "Eat Your Own Dogfood": the honest coverage ledger of the proven-path verifier; a napkin PEM decode (the fixed 12-byte Ed25519 SPKI prefix, read with your eyes) paired with the mechanical extraction; live backend dispatch; the fail-closed --require-verified-verifier policy; and the hybrid-PQC section - proven-classical Ed25519 plus a required-but-honest ML-DSA slot ("blockers get fixed; placeholders get trusted"). Every code cell of the changed notebooks was executed end-to-end before committing (outputs stripped per house rules). 49/49 tests green with the notebook inventory updated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 08:18:06 +00:00
"- Research checkpoint: compare PACTA's pin store to production Certificate Transparency monitor/gossip requirements - what does gossip add that a single pin store cannot?\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"name": "python",
"pygments_lexer": "ipython3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}