proof-aware-crypto-tooling-agent/notebooks/06a_provider_build_the_log.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Lecture 6a: THE PROVIDER'S SIDE - Building the Authenticated Structure\n",
        "\n",
        "> **DOMAIN BANNER - read this first.** In this notebook YOU ARE THE\n",
        "> PROVIDER. There is exactly **one** of you per log. You hold the\n",
        "> signing key. You own a Lean toolchain and hours of compute. You\n",
        "> carry the append-only obligations. Nothing in this notebook is\n",
        "> ever executed by an agent - and that asymmetry is not an\n",
        "> implementation detail, it is the entire design (see the\n",
        "> justification at the end).\n",
        "\n",
        "The provider's job, end to end: **verify -> leaf -> tree -> sign ->\n",
        "self-check**. Only the first step involves Lean; everything after\n",
        "is hashing and one signature.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Learning Objectives\n",
        "\n",
        "- Build the full authenticated data structure from real attestations: leaves, tree, Signed Tree Head.\n",
        "- Place the Lean verification correctly: it is the LEAF-MAKING step, the only expensive one, and it never travels to the agent.\n",
        "- Sign the root with the merkleized library and run the provider's own inclusion self-check (\"the provider eats its own dogfood\").\n",
        "- Justify the singleton/many split as a design decision.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Step 1 - Verify (the expensive step, done ONCE)\n",
        "\n",
        "The leaf content is a signed **attestation**: the outcome of replaying\n",
        "every Lean proof of one repository under lean-guard (~30 minutes of\n",
        "kernel re-checking per fork on the reference machine). This notebook\n",
        "does NOT re-run that - the shipped `evidence/` attestations ARE that\n",
        "step's output. What matters architecturally: **the Lean cost lives\n",
        "here and only here.** No agent ever pays it again.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from pathlib import Path\n",
        "import sys\n",
        "\n",
        "repo_root = Path.cwd()\n",
        "if not (repo_root / \"src\" / \"pacta\").exists():\n",
        "    repo_root = repo_root.parent\n",
        "sys.path.insert(0, str(repo_root / \"src\"))\n",
        "sys.path.insert(0, str(repo_root / \"provider\" / \"src\"))\n",
        "\n",
        "from pacta.yamlio import load_data\n",
        "\n",
        "attestations = {\n",
        "    fork: load_data(repo_root / \"evidence\" / f\"{fork}-ed25519.attestation.yaml\")\n",
        "    for fork in [\"dalek\", \"anza\", \"risc0\", \"betrusted\"]\n",
        "}\n",
        "for fork, att in attestations.items():\n",
        "    certs = att[\"certificates\"]\n",
        "    clean = sum(1 for c in certs if c[\"status\"] == \"proven\" and c[\"axiom_status\"] == \"clean\")\n",
        "    print(f\"{fork}: {clean}/{len(certs)} proven | commit {att['subject']['repo_commit'][:8]} | guard: {att['machine_protection']['lean_guard'].rsplit('/',1)[-1]}\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Steps 2+3 - Leaf and tree (cheap, mechanical)\n",
        "\n",
        "Each attestation is wrapped, canonically serialized, and hashed with\n",
        "the RFC 9162 leaf prefix `0x00`; pairs of nodes hash with prefix\n",
        "`0x01`. Build a REAL provider log in a scratch directory - you are\n",
        "the provider, so mint your own key first:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import tempfile\n",
        "from pacta.signing import generate_ed25519_keypair\n",
        "from pacta_provider.transparency_log import TransparencyLog\n",
        "\n",
        "state = Path(tempfile.mkdtemp(prefix=\"provider-lecture-\"))\n",
        "generate_ed25519_keypair(state / \"provider.key\", state / \"provider.pub\")\n",
        "log = TransparencyLog(state / \"log\")\n",
        "log.init(\"lecture-provider\", state / \"provider.pub\")\n",
        "\n",
        "receipts = {}\n",
        "for fork, att in attestations.items():\n",
        "    att_path = state / f\"{fork}.attestation.yaml\"\n",
        "    from pacta.yamlio import dump_data\n",
        "    dump_data(att, att_path)\n",
        "    receipts[fork] = log.append_attestation(att_path, state / \"provider.key\", state / \"provider.pub\")\n",
        "print(\"tree size:\", receipts[\"betrusted\"][\"tree_size\"])\n",
        "print(\"root:\", receipts[\"betrusted\"][\"sth\"][\"root_hash\"][:32], \"\u2026\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Steps 4+5 - Sign the root, then CHECK YOURSELF\n",
        "\n",
        "The tree head is signed with the **merkleized library itself** (when\n",
        "the dogfood binary is built): the Ed25519 code that signs this root\n",
        "is the same pinned dalek source whose proof attestation is a leaf of\n",
        "this very tree. Before signing, the provider runs the SAME Merkle\n",
        "inclusion verification an agent would run - on its own signing\n",
        "library's leaf, against the tree it is about to sign - and embeds\n",
        "the verdict in the signature block. A root signature that names the\n",
        "leaf vouching for the code that produced it:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import json\n",
        "\n",
        "sth = receipts[\"betrusted\"][\"sth\"]\n",
        "ed = sth[\"signatures\"][\"ed25519\"]\n",
        "print(\"signing backend:\", ed.get(\"signing_backend\"))\n",
        "print(json.dumps(ed.get(\"signing_provenance\", {\"note\": \"dogfood binary not built on this host - OpenSSL fallback, provenance omitted\"}), indent=1))\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## The structure you just built, drawn from your own log\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from pacta.transparency import leaf_hash, merkle_root, node_hash\n",
        "\n",
        "def svg_merkle(leaf_hashes, highlight=None, title=\"\", domain=\"PROVIDER: builds every box below\", color=\"#1e7f4f\"):\n",
        "    n = len(leaf_hashes)\n",
        "    width, lh, lv = 980, 108, 92\n",
        "    levels = []\n",
        "    level = [bytes.fromhex(h) if isinstance(h, str) else h for h in leaf_hashes]\n",
        "    levels.append(level)\n",
        "    while len(level) > 1:\n",
        "        nxt = []\n",
        "        for i in range(0, len(level) - 1, 2):\n",
        "            nxt.append(node_hash(level[i], level[i + 1]))\n",
        "        if len(level) % 2:\n",
        "            nxt.append(level[-1])\n",
        "        levels.append(nxt)\n",
        "        level = nxt\n",
        "    height = 130 + lv * len(levels) + 60\n",
        "    out = [f'<svg xmlns=\"http://www.w3.org/2000/svg\" width=\"{width}\" height=\"{height}\" font-family=\"monospace\" font-size=\"11\">']\n",
        "    out.append(f'<rect x=\"4\" y=\"4\" width=\"{width-8}\" height=\"{height-8}\" fill=\"none\" stroke=\"{color}\" stroke-width=\"2\" rx=\"8\"/>')\n",
        "    out.append(f'<text x=\"16\" y=\"24\" fill=\"{color}\" font-size=\"13\" font-weight=\"bold\">{domain}</text>')\n",
        "    out.append(f'<text x=\"16\" y=\"42\" fill=\"#555\">{title}</text>')\n",
        "    pos = {}\n",
        "    for li, lvl in enumerate(levels):\n",
        "        y = height - 70 - li * lv\n",
        "        span = width / (len(lvl) + 1)\n",
        "        for i, node in enumerate(lvl):\n",
        "            x = span * (i + 1)\n",
        "            pos[(li, i)] = (x, y)\n",
        "            hl = highlight and li == 0 and i == highlight[0]\n",
        "            sib = highlight and (li, i) in highlight[1]\n",
        "            fill = \"#fdf0da\" if sib else (\"#e2f2e9\" if hl else \"#f4f4f6\")\n",
        "            stroke = \"#a86a10\" if sib else (\"#1e7f4f\" if hl else \"#999\")\n",
        "            out.append(f'<rect x=\"{x-52}\" y=\"{y-16}\" width=\"104\" height=\"32\" fill=\"{fill}\" stroke=\"{stroke}\" stroke-width=\"{2 if (hl or sib) else 1}\" rx=\"4\"/>')\n",
        "            label = (\"leaf %d\" % i) if li == 0 else (\"root\" if li == len(levels) - 1 else \"node\")\n",
        "            out.append(f'<text x=\"{x}\" y=\"{y-3}\" text-anchor=\"middle\" fill=\"#333\">{label}</text>')\n",
        "            out.append(f'<text x=\"{x}\" y=\"{y+11}\" text-anchor=\"middle\" fill=\"#777\">{node.hex()[:10]}\u2026</text>')\n",
        "            if li > 0:\n",
        "                for ci in (2 * i, 2 * i + 1):\n",
        "                    if (li - 1, ci) in pos:\n",
        "                        cx, cy = pos[(li - 1, ci)]\n",
        "                        out.append(f'<line x1=\"{x}\" y1=\"{y+16}\" x2=\"{cx}\" y2=\"{cy-16}\" stroke=\"#bbb\"/>')\n",
        "    rx, ry = pos[(len(levels) - 1, 0)]\n",
        "    out.append(f'<rect x=\"{rx-135}\" y=\"{ry-62}\" width=\"270\" height=\"30\" fill=\"#eef\" stroke=\"#3b4d8f\" rx=\"4\"/>')\n",
        "    out.append(f'<text x=\"{rx}\" y=\"{ry-42}\" text-anchor=\"middle\" fill=\"#3b4d8f\">Signed Tree Head: Ed25519(root) via merkleized library</text>')\n",
        "    out.append(f'<line x1=\"{rx}\" y1=\"{ry-32}\" x2=\"{rx}\" y2=\"{ry-16}\" stroke=\"#3b4d8f\"/>')\n",
        "    out.append(\"</svg>\")\n",
        "    return \"\".join(out)\n",
        "\n",
        "entries = log.entries()\n",
        "leaf_hexes = [leaf_hash(e.leaf_bytes()).hex() for e in entries]\n",
        "svg = svg_merkle(leaf_hexes, title=f\"your lecture log: {len(entries)} attestation leaves, root {merkle_root([e.leaf_bytes() for e in entries]).hex()[:16]}\u2026\")\n",
        "try:\n",
        "    from IPython.display import SVG, display\n",
        "    display(SVG(svg))\n",
        "except Exception:\n",
        "    print(svg[:200], \"\u2026 (open in a notebook to render)\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Why a singleton? The design justification\n",
        "\n",
        "| | Provider (this notebook) | Agent (next notebook) |\n",
        "|---|---|---|\n",
        "| How many | **exactly one** per log | unbounded |\n",
        "| Owns | the signing key, the Lean toolchain, the full log | the provider's PUBLIC key, a pin file |\n",
        "| Pays | hours of kernel time per repo, ONCE | milliseconds per check, forever |\n",
        "| Obligations | append-only, sign every head, serve proofs, self-verify | pin every head, demand consistency |\n",
        "| Can be wrong? | detectably: signatures + pins make lies attributable | fails closed |\n",
        "\n",
        "The asymmetry is the product. If agents had to run Lean, the service\n",
        "would add nothing; if the provider's claims weren't pinned and\n",
        "signed, trust would be a rumor. Every artifact in this course lives\n",
        "on exactly one side of this table - and the split between this\n",
        "notebook and the next MIRRORS it on purpose: if you cannot say\n",
        "which notebook a step belongs to, you have not understood the step.\n",
        "\n",
        "## Exercises\n",
        "\n",
        "- Append a fifth attestation (edit one field of a copy) and watch the root change; which internal nodes changed and which did not? Explain from the tree shape.\n",
        "- The self-inclusion check ran against the tree BEFORE your key existed in any leaf. What does `signing_provenance.self_inclusion` say, and why is recording that honest?\n",
        "- Cost accounting: with 4 repos x 30 minutes of Lean and N agents x 5 ms of verification, at what N does the provider model beat every-agent-verifies-locally? (Hint: N=1.)\n",
        "- Design question: what breaks if there are TWO providers with one key? With two keys and one log?\n"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "pygments_lexer": "ipython3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}