proof-aware-crypto-tooling-.../notebooks/06_merkle_transparency_logs.ipynb

235 lines
9.7 KiB
Text
Raw Normal View History

{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Lecture 6: Merkle Transparency Logs\n",
"\n",
"Transparency logs make signed statements auditable. PACTA uses an RFC 9162-style Merkle accumulator over signed proof-check attestations. A provider signs the tree head, and an agent verifies an inclusion proof before acting on the attestation.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Learning Objectives\n",
"\n",
"- Implement leaf and node hashing with domain separation.\n",
"- Compute a Merkle root.\n",
"- Generate and verify inclusion proofs.\n",
"- Generate and verify consistency proofs.\n",
"- Explain Signed Tree Heads and signature policy.\n",
"- Explain why ML-DSA must fail closed when unavailable.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## RFC 9162 Hash Shape\n",
"\n",
"PACTA follows the Certificate Transparency hash structure:\n",
"\n",
"- Empty tree hash: `SHA256(\"\")`\n",
"- Leaf hash: `SHA256(0x00 || leaf_input)`\n",
"- Node hash: `SHA256(0x01 || left || right)`\n",
"\n",
"The prefix bytes prevent a leaf value from being confused with an internal node value.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import sys\n",
"\n",
"repo_root = Path.cwd()\n",
"if not (repo_root / \"src\" / \"pacta\").exists():\n",
" repo_root = repo_root.parent\n",
"sys.path.insert(0, str(repo_root / \"src\"))\n",
"\n",
"from pacta.transparency import (\n",
" leaf_hash,\n",
" node_hash,\n",
" merkle_root,\n",
" inclusion_proof,\n",
" verify_inclusion,\n",
" consistency_proof,\n",
" verify_consistency,\n",
")\n",
"\n",
"leaves = [f\"attestation-{i}\".encode() for i in range(1, 6)]\n",
"root = merkle_root(leaves)\n",
"print(root.hex())\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"for index, leaf in enumerate(leaves):\n",
" proof = inclusion_proof(leaves, index)\n",
" ok = verify_inclusion(leaf, index, len(leaves), proof, root)\n",
" print(index, ok, [node.hex()[:12] for node in proof])\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Consistency Proofs\n",
"\n",
"An inclusion proof answers: \"Is this leaf in this tree?\"\n",
"\n",
"A consistency proof answers: \"Is the newer tree an append-only extension of the older tree?\"\n",
"\n",
"Both are needed for a monitored transparency system. Inclusion is enough for one agent to bind one attestation to one signed tree head. Consistency lets monitors detect equivocation or tree rewrites across time.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"old_size = 3\n",
"old_root = merkle_root(leaves[:old_size])\n",
"new_root = merkle_root(leaves)\n",
"proof = consistency_proof(leaves, old_size)\n",
"print(\"old:\", old_root.hex())\n",
"print(\"new:\", new_root.hex())\n",
"print(\"proof:\", [node.hex()[:12] for node in proof])\n",
"print(\"consistent:\", verify_consistency(old_size, len(leaves), old_root, new_root, proof))\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Signed Tree Heads\n",
"\n",
"A Signed Tree Head records:\n",
"\n",
"- log ID,\n",
"- tree size,\n",
"- timestamp,\n",
"- root hash,\n",
"- hash algorithm,\n",
"- signatures.\n",
"\n",
"PACTA signs the canonical JSON STH payload with Ed25519 through OpenSSL. It also records an ML-DSA-65 slot. On this host, if no real ML-DSA backend is present, the slot is `unavailable`.\n",
"\n",
"Policy matters:\n",
"\n",
"- `require-signatures ed25519`: verify Ed25519 and allow ML-DSA to be unavailable.\n",
"- `require-signatures both`: require Ed25519 and ML-DSA verified. If ML-DSA is unavailable, fail closed.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from pacta.postquantum import detect_ml_dsa\n",
"\n",
"capability = detect_ml_dsa()\n",
"print(capability.available)\n",
"print(capability.backend)\n",
"print(capability.reason)\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Why Ed25519 and ML-DSA Together?\n",
"\n",
"Ed25519 is useful because it is widely deployed, fast, and directly relevant to the Ed25519 proof corpus. That creates a deliberate \"eat your own dogfood\" loop: the proof-checking ecosystem signs evidence using a primitive whose implementation family is under formal scrutiny.\n",
"\n",
"ML-DSA adds post-quantum robustness for the accumulator signature layer. But it must be a real signature, not an aspirational label. If a host lacks ML-DSA, the correct result is an explicit blocker.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
Curriculum: the ratchet rule, the four-tier reality, and lecture 9 (dogfood) The notebooks now carry the same didactic contract as the companion book (the "ratchet rule", stated in the course map): every load-bearing idea runs twice - napkin scale, then real scale - and every pair is EXECUTABLE in the notebook, not narrated. - Lecture 1: the truth boundary updated to the proven four-tier apex, with the what-is-still-NOT-proven list (SHA-512, parsers, signing, wallets) given equal weight. - Lecture 2: napkin/real scoring pair - a two-certificate toy card scored in your head, then the shipped sixteen-certificate R4 fixture through the same function, residual blockers and per-tier boundary axioms printed. - Lecture 6: new split-view section. A runnable equivocation drill: pin a two-leaf view, grow it honestly with a consistency proof, then present a forged same-size root and watch the pin store name the attack. Real-scale pointers to --sth-store, log-consistency, log-audit, and the freshness policy; a new exercise asks students to construct the lie a size-only anchor check would miss. - Lecture 7: the wallet gate now swings BOTH ways on real evidence - a partial card denied at R3, the shipped R4 card allowed - both runnable. - Lecture 8 capstone: "design R4" became "audit R4": read the shipped card like an auditor, then design the R5 discharge plan (parser specs, verified SHA-512, signing-side, per-fork production-path mapping). - NEW Lecture 9, "Eat Your Own Dogfood": the honest coverage ledger of the proven-path verifier; a napkin PEM decode (the fixed 12-byte Ed25519 SPKI prefix, read with your eyes) paired with the mechanical extraction; live backend dispatch; the fail-closed --require-verified-verifier policy; and the hybrid-PQC section - proven-classical Ed25519 plus a required-but-honest ML-DSA slot ("blockers get fixed; placeholders get trusted"). Every code cell of the changed notebooks was executed end-to-end before committing (outputs stripped per house rules). 49/49 tests green with the notebook inventory updated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 08:18:06 +00:00
"## Split Views: why a receipt is not enough\n",
"\n",
"Everything above verifies ONE receipt against ONE signed tree head. A malicious provider can maintain TWO trees - one shown to you, one shown to the world - and both views verify perfectly in isolation. This is EQUIVOCATION, and the defense is memory: pin every tree head you accept, and demand that every later tree head be CONSISTENT with your pin (same size -> same root; larger size -> a verified consistency proof from your pinned size; smaller size -> rollback, reject forever).\n",
"\n",
"pacta implements this as a local STH pin store. Run the whole attack and its detection, napkin-size:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# NAPKIN: pin a 2-leaf view, then let the log grow honestly - and then\n",
"# let a SPLIT VIEW present a different root at the pinned size.\n",
"import tempfile\n",
"from pathlib import Path as _P\n",
"from pacta.sthstore import check_sth_against_store\n",
"from pacta.transparency import consistency_proof, merkle_root, proof_to_hex\n",
"\n",
"honest = [b\"attestation-A\", b\"attestation-B\", b\"attestation-C\"]\n",
"evil = [b\"attestation-A\", b\"attestation-EVIL\", b\"attestation-C\"]\n",
"\n",
"with tempfile.TemporaryDirectory() as tmp:\n",
" store = _P(tmp) / \"sth-store.json\"\n",
" sth = lambda size, leaves: {\n",
" \"log_id\": \"demo-log\", \"tree_size\": size,\n",
" \"root_hash\": merkle_root(leaves[:size]).hex(),\n",
" \"timestamp\": \"2026-07-06T00:00:00Z\",\n",
" }\n",
" print(\"pin: \", check_sth_against_store(sth(2, honest), store).diagnostics[0])\n",
" grown = check_sth_against_store(\n",
" sth(3, honest), store,\n",
" consistency_proof_hex=proof_to_hex(consistency_proof(honest, 2)),\n",
" )\n",
" print(\"grow: \", grown.diagnostics[0])\n",
" attack = check_sth_against_store(sth(3, evil), store)\n",
" print(\"attack ok?\", attack.ok)\n",
" print(\"verdict: \", attack.diagnostics[0][:120], \"...\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"At real scale the same check runs on every `pacta receipt-verify --sth-store ...` and `pacta agent --sth-store ...` invocation; receipts embed a consistency anchor from the previous tree size, the provider serves proofs from arbitrary pinned sizes (`pacta_provider log-consistency --from-size N`), and `pacta_provider log-audit` is the monitor's self-check. A freshness policy (`--max-sth-age-seconds`) closes the stale-root hole: an old-but-valid tree head could hide later entries.\n",
"\n",
"## Exercises\n",
"\n",
"- Tamper with one leaf and show that inclusion verification fails.\n",
"- Explain why the tree head signature must cover tree size as well as root hash.\n",
Curriculum: the ratchet rule, the four-tier reality, and lecture 9 (dogfood) The notebooks now carry the same didactic contract as the companion book (the "ratchet rule", stated in the course map): every load-bearing idea runs twice - napkin scale, then real scale - and every pair is EXECUTABLE in the notebook, not narrated. - Lecture 1: the truth boundary updated to the proven four-tier apex, with the what-is-still-NOT-proven list (SHA-512, parsers, signing, wallets) given equal weight. - Lecture 2: napkin/real scoring pair - a two-certificate toy card scored in your head, then the shipped sixteen-certificate R4 fixture through the same function, residual blockers and per-tier boundary axioms printed. - Lecture 6: new split-view section. A runnable equivocation drill: pin a two-leaf view, grow it honestly with a consistency proof, then present a forged same-size root and watch the pin store name the attack. Real-scale pointers to --sth-store, log-consistency, log-audit, and the freshness policy; a new exercise asks students to construct the lie a size-only anchor check would miss. - Lecture 7: the wallet gate now swings BOTH ways on real evidence - a partial card denied at R3, the shipped R4 card allowed - both runnable. - Lecture 8 capstone: "design R4" became "audit R4": read the shipped card like an auditor, then design the R5 discharge plan (parser specs, verified SHA-512, signing-side, per-fork production-path mapping). - NEW Lecture 9, "Eat Your Own Dogfood": the honest coverage ledger of the proven-path verifier; a napkin PEM decode (the fixed 12-byte Ed25519 SPKI prefix, read with your eyes) paired with the mechanical extraction; live backend dispatch; the fail-closed --require-verified-verifier policy; and the hybrid-PQC section - proven-classical Ed25519 plus a required-but-honest ML-DSA slot ("blockers get fixed; placeholders get trusted"). Every code cell of the changed notebooks was executed end-to-end before committing (outputs stripped per house rules). 49/49 tests green with the notebook inventory updated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 08:18:06 +00:00
"- Napkin, then real: run the split-view drill above; then initialize a real provider log (`pacta_provider log-init`), append two attestations, and verify the second receipt with `--sth-store` - watch the pin advance with a verified consistency proof.\n",
"- Why must the consistency anchor's ROOT (not just its size) be checked against the pin? Construct the lie that a size-only check would miss.\n",
"- Write a policy for when an autonomous agent should require `both` signatures.\n",
Curriculum: the ratchet rule, the four-tier reality, and lecture 9 (dogfood) The notebooks now carry the same didactic contract as the companion book (the "ratchet rule", stated in the course map): every load-bearing idea runs twice - napkin scale, then real scale - and every pair is EXECUTABLE in the notebook, not narrated. - Lecture 1: the truth boundary updated to the proven four-tier apex, with the what-is-still-NOT-proven list (SHA-512, parsers, signing, wallets) given equal weight. - Lecture 2: napkin/real scoring pair - a two-certificate toy card scored in your head, then the shipped sixteen-certificate R4 fixture through the same function, residual blockers and per-tier boundary axioms printed. - Lecture 6: new split-view section. A runnable equivocation drill: pin a two-leaf view, grow it honestly with a consistency proof, then present a forged same-size root and watch the pin store name the attack. Real-scale pointers to --sth-store, log-consistency, log-audit, and the freshness policy; a new exercise asks students to construct the lie a size-only anchor check would miss. - Lecture 7: the wallet gate now swings BOTH ways on real evidence - a partial card denied at R3, the shipped R4 card allowed - both runnable. - Lecture 8 capstone: "design R4" became "audit R4": read the shipped card like an auditor, then design the R5 discharge plan (parser specs, verified SHA-512, signing-side, per-fork production-path mapping). - NEW Lecture 9, "Eat Your Own Dogfood": the honest coverage ledger of the proven-path verifier; a napkin PEM decode (the fixed 12-byte Ed25519 SPKI prefix, read with your eyes) paired with the mechanical extraction; live backend dispatch; the fail-closed --require-verified-verifier policy; and the hybrid-PQC section - proven-classical Ed25519 plus a required-but-honest ML-DSA slot ("blockers get fixed; placeholders get trusted"). Every code cell of the changed notebooks was executed end-to-end before committing (outputs stripped per house rules). 49/49 tests green with the notebook inventory updated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 08:18:06 +00:00
"- Research checkpoint: compare PACTA's pin store to production Certificate Transparency monitor/gossip requirements - what does gossip add that a single pin store cannot?\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"name": "python",
"pygments_lexer": "ipython3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}