betrusted-ed25519-verified/verification/model-correspondence.py

#!/usr/bin/env python3
"""Classify every external the extraction declares.

For each gen/<dir>/<X>_Template.lean, Aeneas states what the extracted Rust
needs from outside. Each such name must be provided by exactly one of:

  MODEL   — declared in the hand-written sibling gen/<dir>/<X>.lean: an
            assumption, which the axiom gate and the per-certificate cones
            then govern;
  PROVEN  — resolved to a real definition in the proven corpus, because a
            module of this repository declares it (namespace-aware).

Anything else is drift: the extraction asks for something this repository does
not provide.

────────────────────────────────────────────────────────────────────────────
WHY THIS FILE WAS REWRITTEN — 2026-08-01, round-7 external review

The first version matched declarations with a LINE-ORIENTED regex requiring the
keyword and the name on the same physical line, and it did not strip comments.
Both assumptions are false about Lean, and false about Aeneas's own output.
Three of the four forks contain, verbatim:

    axiom
      curve25519_dalek.edwards.EdwardsPoint.Insts.CoreOpsArithNegEdwardsPoint.neg
      :
      curve25519_dalek.edwards.EdwardsPoint -> Result ...

The old pattern matched nothing there, so that declaration was SILENTLY
DROPPED: no MODEL row, no PROVEN row, and no failure. Every committed
MODEL-CORRESPONDENCE.txt was missing it, and every button passed green over the
incomplete table. A reviewer separately showed that a definition appearing only
inside a `/- ... -/` comment was read as a real declaration, so the scanner
could also report PROVEN for a name Lean resolves to an axiom.

The lesson is not "write a better regex". It is that this scanner was
FAIL-OPEN: input it could not parse produced silence instead of a stop. A gate
that drops what it cannot read is worse than no gate, because the button prints
green across the gap and the gap is invisible in the diff.

This version therefore:
  · strips comments first, including NESTED `/- ... -/` blocks, which Lean has
    and which a non-greedy match would close at the first inner `-/`;
  · allows a declaration's name to appear on a later line than its keyword;
  · tracks `namespace` / `section` / `end` over the stripped text;
  · FAILS CLOSED — every declaration keyword must yield a name, or the scanner
    exits non-zero naming file and line. Nothing is dropped, ever.

WHAT IT STILL IS NOT. This is a source scanner, not a semantic Lean query. It
cannot see `export`, aliases, or how Lean actually resolves a name at
elaboration. A PROVEN row is documentary evidence about the extraction
boundary; it is NOT a Lean-checked fact, and the trust documents must not claim
it is. What the estate relies on for soundness is kernel-side and
environment-derived — Phase 2b's axiom gate, Phase 2c's inventory, and the
exact per-certificate cones of Phase 3/3b — none of which consult this file.
────────────────────────────────────────────────────────────────────────────
"""
import re
import sys
import os
import glob

KEYWORDS = ('axiom', 'def', 'abbrev', 'opaque', 'structure', 'inductive',
            'instance', 'theorem', 'lemma')

# A declaration keyword opening a logical line, after any attributes and
# modifiers. The NAME is deliberately NOT part of this pattern: it may sit on a
# later line, which is precisely the case the previous scanner dropped.
KW = re.compile(
    r'^[ \t]*(?:@\[[^\]]*\][ \t\n]*)*'
    r'(?:private |protected |noncomputable |unsafe |partial |scoped |local )*'
    r'(' + '|'.join(KEYWORDS) + r')(?=[ \t\n])',
    re.M)

IDENT = re.compile(r"[ \t\n]*([A-Za-z_][A-Za-z0-9_.'!?]*)")

NS = re.compile(
    r"^[ \t]*(namespace|section|end)(?:[ \t]+([A-Za-z_][A-Za-z0-9_.']*))?[ \t]*$",
    re.M)


class ScanError(Exception):
    """Raised when a declaration cannot be parsed. Never swallowed."""


def strip_comments(text):
    """Remove Lean comments, preserving newlines so line numbers stay true.

    Block comments NEST in Lean, so this needs a depth counter: a non-greedy
    `/-.*?-/` would close the outer block at the first inner `-/` and leave the
    tail of a nested comment looking like source.
    """
    out, i, n, depth = [], 0, len(text), 0
    while i < n:
        if text.startswith('/-', i):
            depth += 1
            out.append('  ')
            i += 2
            continue
        if text.startswith('-/', i):
            if depth:
                depth -= 1
            out.append('  ')
            i += 2
            continue
        if depth:
            out.append('\n' if text[i] == '\n' else ' ')
            i += 1
            continue
        if text.startswith('--', i):
            j = text.find('\n', i)
            if j < 0:
                out.append(' ' * (n - i))
                break
            out.append(' ' * (j - i))
            i = j
            continue
        out.append(text[i])
        i += 1
    return ''.join(out)


def declared(path):
    """Fully-qualified names declared in one file.

    Raises ScanError on any declaration keyword whose name cannot be read.
    """
    raw = open(path, encoding='utf-8', errors='replace').read()
    text = strip_comments(raw)

    # Scope events by offset, so each declaration can be placed in its stack.
    events = [(m.start(), m.group(1), m.group(2)) for m in NS.finditer(text)]

    names = set()
    for m in KW.finditer(text):
        im = IDENT.match(text, m.end())
        if not im:
            line = text.count('\n', 0, m.start()) + 1
            raise ScanError(
                "%s:%d: `%s` with no parseable name. This scanner fails closed:"
                " it will not drop a declaration it cannot read."
                % (path, line, m.group(1)))
        stack = []
        for off, kind, arg in events:
            if off > m.start():
                break
            if kind in ('namespace', 'section'):
                stack.append(arg)
            elif stack:
                stack.pop()
        prefix = [p for p in stack if p]
        names.add('.'.join(prefix + [im.group(1)]) if prefix else im.group(1))
    return names


def main(root):
    gen = os.path.join(root, 'gen')
    templates = sorted(glob.glob(os.path.join(gen, '*', '*_Template.lean')))
    # The proven corpus: every generated module that is neither a template nor
    # a hand-written model. These are the files Aeneas produced from Rust.
    models = {t.replace('_Template', '') for t in templates}
    corpus = set()
    for f in sorted(glob.glob(os.path.join(gen, '*', '*.lean'))):
        if f in models or f.endswith('_Template.lean'):
            continue
        corpus |= declared(f)

    rows, unresolved = [], []
    for t in templates:
        model = t.replace('_Template', '')
        rel = os.path.relpath(t, gen).replace('_Template.lean', '')
        tnames = declared(t)
        mnames = declared(model) if os.path.exists(model) else set()
        for n in sorted(tnames):
            if n in mnames:
                rows.append(f'{rel}|{n}|MODEL')
            elif n in corpus:
                rows.append(f'{rel}|{n}|PROVEN')
            else:
                rows.append(f'{rel}|{n}|UNRESOLVED')
                unresolved.append(f'{rel}|{n}')
        for n in sorted(mnames - tnames):
            rows.append(f'{rel}|{n}|EXTRA')
    print('\n'.join(rows))
    print(f'CORRESPONDENCE-COUNT|{len(rows)}')
    return 1 if unresolved else 0


def emit_names(root):
    """Every name the EXTRACTION asks for, as `<rel>|<name>`.

    Template discovery is unavoidably textual: the template is not imported (it
    would clash with the model, which declares the same names), so no Lean
    environment contains it. That is why `declared()` fails closed — this list
    is the input to the semantic phase, and a name missing here is a name
    nothing will ever check.
    """
    gen = os.path.join(root, 'gen')
    for t in sorted(glob.glob(os.path.join(gen, '*', '*_Template.lean'))):
        rel = os.path.relpath(t, gen).replace('_Template.lean', '')
        for n in sorted(declared(t)):
            print(f'{rel}|{n}')
    return 0


if __name__ == '__main__':
    try:
        if len(sys.argv) > 2 and sys.argv[1] == '--names':
            sys.exit(emit_names(sys.argv[2]))
        sys.exit(main(sys.argv[1]))
    except ScanError as e:
        # Fail closed and loudly. Never degrade to a partial table.
        print('MODEL CORRESPONDENCE SCAN FAILED: %s' % e, file=sys.stderr)
        sys.exit(2)