{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Track C: The Search — Optimization and the Ratchet\n",
    "\n",
    "**Plan C — Parallel Tracks**\n",
    "\n",
    "This track covers automated parameter search. You will learn how the ratchet generates challengers, selects winners, extracts lessons, and narrows the search space across rungs.\n",
    "\n",
    "> **Dashboard:** Use `00_dashboard.ipynb` to explore individual experiments manually, then see how the ratchet does it automatically."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "%matplotlib inline\n",
    "import warnings, tempfile\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from math import sqrt\n",
    "\n",
    "from autoresearch_quantum.models import (\n",
    "    ExperimentSpec, RungConfig, EvaluationMetrics,\n",
    "    QualityWeights, CostWeights, ScoreConfig, SearchSpaceConfig,\n",
    "    TierPolicyConfig, HardwareConfig, LessonFeedback, SearchRule,\n",
    ")\n",
    "from autoresearch_quantum.execution.local import LocalCheapExecutor\n",
    "from autoresearch_quantum.search.challengers import (\n",
    "    generate_neighbor_challengers, mutation_summary, GeneratedChallenger,\n",
    ")\n",
    "from autoresearch_quantum.search.strategies import (\n",
    "    NeighborWalk, RandomCombo, LessonGuided, CompositeGenerator,\n",
    "    default_composite, StrategyWeight,\n",
    ")\n",
    "from autoresearch_quantum.ratchet.runner import AutoresearchHarness\n",
    "from autoresearch_quantum.persistence.store import ResearchStore\n",
    "from autoresearch_quantum.config import load_rung_config\n",
    "from autoresearch_quantum.lessons.extractor import extract_rung_lesson\n",
    "from autoresearch_quantum.lessons.feedback import (\n",
    "    extract_search_rules, narrow_search_space, build_lesson_feedback,\n",
    ")\n",
    "from autoresearch_quantum.execution.transfer import TransferEvaluator\n",
    "from matplotlib.patches import Patch\n",
    "\n",
    "print(\"All imports successful.\")"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "from autoresearch_quantum.teaching import LearningTracker\n",
    "from autoresearch_quantum.teaching.assess import quiz, predict_choice, reflect, order, checkpoint_summary\n",
    "tracker = LearningTracker(\"plan_c_track_c\")\n",
    "print(\"Learning tracker active.\")"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## 1. The Parameter Space\n",
    "\n",
    "The rung1 config defines a discrete search space over circuit parameters."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "rung_config = load_rung_config(\"../../configs/rungs/rung1.yaml\")\n",
    "\n",
    "print(f\"Rung: {rung_config.name}\")\n",
    "print(f\"Objective: {rung_config.objective}\")\n",
    "print(f\"\\nSearch dimensions:\")\n",
    "total_combos = 1\n",
    "for dim, values in rung_config.search_space.dimensions.items():\n",
    "    print(f\"  {dim:25s}: {values}\")\n",
    "    total_combos *= len(values)\n",
    "print(f\"\\nTotal combinations: {total_combos}\")\n",
    "print(f\"Max challengers per step: {rung_config.search_space.max_challengers_per_step}\")\n",
    "print(f\"Step budget: {rung_config.step_budget}\")\n",
    "print(f\"Patience: {rung_config.patience}\")"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "quiz(tracker, \"q1_why_search\",\n",
    "    question=\"The parameter space is finite. Why not just try every combination?\",\n",
    "    options=[\n",
    "        \"The space is infinite\",\n",
    "        \"Each evaluation costs time/compute; smart search finds good solutions faster\",\n",
    "        \"Exhaustive search always finds worse solutions\",\n",
    "    ],\n",
    "    correct=1, section=\"1. Parameter space\", bloom=\"understand\",\n",
    "    explanation=\"Each evaluation requires noisy simulation (or hardware QPU time). Smart search finds good solutions in fewer evaluations.\")"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With {total} combinations, exhaustive search is feasible but slow. The ratchet is smarter: it starts from a good baseline and explores *neighborhoods*, focusing compute where it matters.\n",
    "\n",
    "---\n",
    "## 2. The Incumbent-Challenger Model\n",
    "\n",
    "Think of it like a chess championship:\n",
    "- The **incumbent** is the reigning champion (best configuration found so far)\n",
    "- **Challengers** are generated by mutating the incumbent's parameters\n",
    "- Each challenger is evaluated (\"plays a match\")\n",
    "- If a challenger beats the incumbent by a sufficient margin, it takes the title\n",
    "\n",
    "This is a form of **local search** — like hill climbing in a discrete parameter space."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "incumbent = rung_config.bootstrap_incumbent\n",
    "print(\"Bootstrap incumbent:\")\n",
    "for field in [\"seed_style\", \"encoder_style\", \"verification\", \"postselection\",\n",
    "              \"ancilla_strategy\", \"optimization_level\"]:\n",
    "    print(f\"  {field:25s}: {getattr(incumbent, field)}\")"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "quiz(tracker, \"q2_incumbent\",\n",
    "    question=\"What is the bootstrap incumbent?\",\n",
    "    options=[\n",
    "        \"A randomly chosen starting point\",\n",
    "        \"A hand-picked reasonable default that the ratchet tries to beat\",\n",
    "        \"The theoretically optimal configuration\",\n",
    "    ],\n",
    "    correct=1, section=\"2. Incumbent\", bloom=\"remember\",\n",
    "    explanation=\"The bootstrap incumbent is a domain-expert guess. The ratchet guarantee: it never gets worse from here.\")\n",
    "checkpoint_summary(tracker, \"2. Incumbent\")"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## 3. NeighborWalk: Single-Axis Perturbation\n",
    "\n",
    "The simplest strategy: change **one parameter at a time** and try all alternatives."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "challengers = generate_neighbor_challengers(incumbent, rung_config.search_space)\n",
    "\n",
    "print(f\"Generated {len(challengers)} challengers (max {rung_config.search_space.max_challengers_per_step}):\\n\")\n",
    "for i, c in enumerate(challengers):\n",
    "    print(f\"  {i+1:2d}. {c.mutation_note}\")"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "quiz(tracker, \"q3_neighborwalk\",\n",
    "    question=\"NeighborWalk changes how many parameters per challenger?\",\n",
    "    options=[\"0\", \"Exactly 1\", \"Up to 3\", \"All of them\"],\n",
    "    correct=1, section=\"3. NeighborWalk\", bloom=\"understand\",\n",
    "    explanation=\"One parameter at a time, trying all alternative values. Systematic but blind to parameter interactions.\")"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Key Insight:** NeighborWalk is exhaustive within single axes but never tests *combinations*. It is fast and deterministic — good for identifying which individual parameter matters most.\n",
    "\n",
    "---\n",
    "## 4. RandomCombo: Multi-Axis Perturbation\n",
    "\n",
    "Change 1-3 parameters simultaneously."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "combo = RandomCombo(num_candidates=8, max_mutations=3)\n",
    "combo_challengers = combo.generate(incumbent, rung_config.search_space, set())\n",
    "\n",
    "print(f\"Generated {len(combo_challengers)} random combo challengers:\\n\")\n",
    "for i, c in enumerate(combo_challengers):\n",
    "    # Count how many fields changed\n",
    "    n_changes = sum(1 for f in incumbent.__dataclass_fields__\n",
    "                    if getattr(incumbent, f) != getattr(c.spec, f))\n",
    "    print(f\"  {i+1:2d}. [{n_changes} changes] {c.mutation_note}\")"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "order(tracker, \"q4_strategy_interactions\",\n",
    "    instruction=\"Rank strategies by ability to find multi-parameter interactions (worst to best):\",\n",
    "    items=[\"NeighborWalk\", \"RandomCombo\"],\n",
    "    correct_order=[\"NeighborWalk\", \"RandomCombo\"],\n",
    "    section=\"4. RandomCombo\", bloom=\"analyze\",\n",
    "    explanation=\"NeighborWalk: 1 axis only, cannot find interactions. RandomCombo mutates multiple axes simultaneously.\")\n",
    "checkpoint_summary(tracker, \"4. RandomCombo\")"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Key Insight:** RandomCombo can discover *interaction effects* — parameter combinations that are better or worse than the sum of individual effects. It introduces diversity but is less systematic.\n",
    "\n",
    "---\n",
    "## 5. Evaluating: Incumbent vs Challengers\n",
    "\n",
    "Let us actually run the experiments and see scores."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "# Use fast settings\n",
    "fast_rung = RungConfig(\n",
    "    rung=1, name=rung_config.name, description=rung_config.description,\n",
    "    objective=rung_config.objective, bootstrap_incumbent=incumbent,\n",
    "    search_space=rung_config.search_space,\n",
    "    tier_policy=TierPolicyConfig(\n",
    "        cheap_margin=0.002, cheap_shots=256, cheap_repeats=1,\n",
    "        expensive_shots=512, expensive_repeats=1,\n",
    "        promote_top_k=2, enable_hardware=False,\n",
    "    ),\n",
    "    score=rung_config.score,\n",
    "    step_budget=1, patience=1, hardware=HardwareConfig(),\n",
    ")\n",
    "\n",
    "executor = LocalCheapExecutor()\n",
    "inc_result = executor.evaluate(incumbent, fast_rung)\n",
    "print(f\"Incumbent score: {inc_result.score:.4f}\\n\")\n",
    "\n",
    "# Evaluate neighbor challengers\n",
    "scores = {}\n",
    "for c in challengers[:8]:\n",
    "    result = executor.evaluate(c.spec, fast_rung)\n",
    "    scores[c.mutation_note] = result.score\n",
    "    marker = \">>>\" if result.score > inc_result.score else \"   \"\n",
    "    print(f\"  {marker} {c.mutation_note[:50]:50s} score={result.score:.4f}\")"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "# Visualize\n",
    "fig, ax = plt.subplots(figsize=(12, 5))\n",
    "labels = [\"INCUMBENT\"] + [k[:35] for k in scores.keys()]\n",
    "vals = [inc_result.score] + list(scores.values())\n",
    "colors = [\"#e74c3c\"] + [\"#2ecc71\" if s > inc_result.score else \"#bdc3c7\" for s in scores.values()]\n",
    "\n",
    "ax.barh(range(len(labels)), vals, color=colors)\n",
    "ax.set_yticks(range(len(labels)))\n",
    "ax.set_yticklabels(labels, fontsize=8)\n",
    "ax.axvline(x=inc_result.score, color=\"#e74c3c\", linestyle=\"--\", alpha=0.5)\n",
    "ax.set_xlabel(\"Score\")\n",
    "ax.set_title(\"Incumbent vs Challengers\")\n",
    "ax.legend(handles=[Patch(color=\"#e74c3c\", label=\"Incumbent\"),\n",
    "                   Patch(color=\"#2ecc71\", label=\"Beats incumbent\"),\n",
    "                   Patch(color=\"#bdc3c7\", label=\"Below incumbent\")])\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## 6. One Ratchet Step in Detail\n",
    "\n",
    "Now let the harness orchestrate everything: generate challengers, evaluate, promote, select winner."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "store = ResearchStore(tempfile.mkdtemp())\n",
    "harness = AutoresearchHarness(store)\n",
    "\n",
    "step = harness.run_ratchet_step(fast_rung, allow_hardware=False)\n",
    "\n",
    "print(f\"Step index:          {step.step_index}\")\n",
    "print(f\"Incumbent before:    {step.incumbent_before_id}\")\n",
    "print(f\"Challengers tested:  {len(step.challengers_tested)}\")\n",
    "print(f\"Promoted:            {len(step.promoted_challengers)}\")\n",
    "print(f\"Winner:              {step.winner_id}\")\n",
    "print(f\"Winning margin:      {step.winning_margin:+.4f}\")\n",
    "print(f\"\\nCheap-tier: {step.cheap_tier_justification}\")\n",
    "print(f\"\\nLesson: {step.distilled_lesson}\")"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "quiz(tracker, \"q5_no_winner\",\n",
    "    question=\"What happens if no challenger beats the incumbent?\",\n",
    "    options=[\n",
    "        \"The harness picks the best challenger anyway\",\n",
    "        \"The incumbent stays; the step is logged with zero improvement\",\n",
    "        \"The harness doubles the number of challengers\",\n",
    "    ],\n",
    "    correct=1, section=\"6. Ratchet step\", bloom=\"understand\",\n",
    "    explanation=\"Ratchet guarantee: the incumbent never gets worse. No-improvement steps are still valuable data for lesson extraction.\")"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## 7. Running a Full Rung with Patience\n",
    "\n",
    "A rung runs multiple steps. **Patience** stops early if no improvement is found."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "store2 = ResearchStore(tempfile.mkdtemp())\n",
    "harness2 = AutoresearchHarness(store2)\n",
    "\n",
    "multi_rung = RungConfig(\n",
    "    rung=1, name=\"Search Demo\", description=\"Full rung demo\",\n",
    "    objective=\"Find best config\",\n",
    "    bootstrap_incumbent=ExperimentSpec(\n",
    "        rung=1, target_backend=\"fake_brisbane\", noise_backend=\"fake_brisbane\",\n",
    "        shots=256, repeats=1,\n",
    "    ),\n",
    "    search_space=SearchSpaceConfig(\n",
    "        dimensions={\n",
    "            \"verification\": [\"both\", \"z_only\", \"x_only\"],\n",
    "            \"seed_style\": [\"h_p\", \"ry_rz\", \"u_magic\"],\n",
    "            \"postselection\": [\"all_measured\", \"z_only\", \"none\"],\n",
    "        },\n",
    "        max_challengers_per_step=6,\n",
    "    ),\n",
    "    tier_policy=TierPolicyConfig(\n",
    "        cheap_margin=0.001, cheap_shots=256, cheap_repeats=1,\n",
    "        promote_top_k=2, enable_hardware=False,\n",
    "    ),\n",
    "    score=rung_config.score,\n",
    "    step_budget=3, patience=2,\n",
    "    hardware=HardwareConfig(),\n",
    ")\n",
    "\n",
    "steps, lesson, feedback = harness2.run_rung(multi_rung, allow_hardware=False)\n",
    "\n",
    "print(f\"Steps completed: {len(steps)}\")\n",
    "for s in steps:\n",
    "    improved = \"IMPROVED\" if s.winning_margin > 0 else \"no change\"\n",
    "    print(f\"  Step {s.step_index}: margin={s.winning_margin:+.4f}  ({improved})\")"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "quiz(tracker, \"q6_patience\",\n",
    "    question=\"Patience=2 means the rung stops after 2 consecutive steps with no improvement. Why?\",\n",
    "    options=[\n",
    "        \"To save memory\",\n",
    "        \"If 2 rounds of challengers all lose, the nearby parameter space is likely exhausted\",\n",
    "        \"2 is always the optimal patience value\",\n",
    "    ],\n",
    "    correct=1, section=\"7. Full rung\", bloom=\"evaluate\",\n",
    "    explanation=\"Patience prevents wasting compute once the search has converged. The budget is better spent on the next rung.\")\n",
    "checkpoint_summary(tracker, \"7. Full rung\")"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "# Score trajectory\n",
    "experiments = store2.list_experiments(1)\n",
    "exp_data = [(e[\"experiment_id\"][:20], e[\"final_score\"], e[\"role\"]) for e in experiments]\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(12, 4))\n",
    "x = range(len(exp_data))\n",
    "colors = [\"#e74c3c\" if role == \"incumbent\" else \"#3498db\" for _, _, role in exp_data]\n",
    "ax.bar(x, [s for _, s, _ in exp_data], color=colors)\n",
    "ax.set_xlabel(\"Experiment\")\n",
    "ax.set_ylabel(\"Score\")\n",
    "ax.set_title(\"All Experiments in Rung\")\n",
    "ax.legend(handles=[Patch(color=\"#e74c3c\", label=\"Incumbent\"), Patch(color=\"#3498db\", label=\"Challenger\")])\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## 8. Lesson Extraction\n",
    "\n",
    "After a rung, the harness analyzes all experiments and extracts **lessons** — both human-readable narratives and machine-readable rules."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "print(\"=\" * 60)\n",
    "print(lesson.narrative)\n",
    "print(\"=\" * 60)"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "reflect(tracker, \"q7_lesson_quality\",\n",
    "    question=\"Read the lesson narrative. What actionable insight does it give? What would make it better?\",\n",
    "    section=\"8. Lessons\", bloom=\"evaluate\",\n",
    "    model_answer=\"A good lesson names specific values that helped/hurt and explains WHY. Machine-readable SearchRules are often more actionable than the narrative.\")"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "print(f\"\\nMachine-readable rules ({len(feedback.rules)}):\\n\")\n",
    "for rule in feedback.rules:\n",
    "    print(f\"  {rule.action.upper():7s} {rule.dimension}={rule.value}\")\n",
    "    print(f\"          confidence={rule.confidence:.2f}  reason: {rule.reason}\\n\")"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "quiz(tracker, \"q8_fix_vs_avoid\",\n",
    "    question=\"'fix' vs 'avoid' rules: what's the difference?\",\n",
    "    options=[\n",
    "        \"'fix' locks a value permanently; 'avoid' removes a value from the search space\",\n",
    "        \"'fix' repairs a bug; 'avoid' prevents a crash\",\n",
    "        \"They are synonyms\",\n",
    "    ],\n",
    "    correct=0, section=\"8. Lessons\", bloom=\"remember\",\n",
    "    explanation=\"'fix': always use this value. 'avoid': never use this value. Both narrow the search space for future rungs.\")\n",
    "checkpoint_summary(tracker, \"8. Lessons\")"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## 9. LessonGuided Strategy\n",
    "\n",
    "Once we have rules, the **LessonGuided** strategy uses them to bias challenger generation toward promising regions."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "if feedback.rules:\n",
    "    guided = LessonGuided(num_candidates=6)\n",
    "    guided_challengers = guided.generate(\n",
    "        incumbent, multi_rung.search_space, set(), [feedback]\n",
    "    )\n",
    "    print(f\"Lesson-guided: {len(guided_challengers)} challengers\")\n",
    "    for c in guided_challengers:\n",
    "        print(f\"  {c.mutation_note}\")\n",
    "else:\n",
    "    print(\"No rules extracted — try a larger step_budget for more data.\")"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## 10. Search Space Narrowing\n",
    "\n",
    "Rules can **narrow** the search space: remove \"avoid\" values, lock \"fix\" values."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "print(\"BEFORE narrowing:\")\n",
    "for dim, vals in multi_rung.search_space.dimensions.items():\n",
    "    print(f\"  {dim}: {vals}\")\n",
    "\n",
    "narrowed = narrow_search_space(multi_rung.search_space, feedback.rules)\n",
    "\n",
    "print(\"\\nAFTER narrowing:\")\n",
    "for dim, vals in narrowed.dimensions.items():\n",
    "    removed = set(multi_rung.search_space.dimensions[dim]) - set(vals)\n",
    "    suffix = f\"  (removed: {removed})\" if removed else \"\"\n",
    "    print(f\"  {dim}: {vals}{suffix}\")"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "quiz(tracker, \"q9_narrowing\",\n",
    "    question=\"What does search space narrowing accomplish?\",\n",
    "    options=[\n",
    "        \"It removes entire parameter dimensions\",\n",
    "        \"It removes poorly-performing values, keeping the dimension with fewer options\",\n",
    "        \"It adds new parameter values\",\n",
    "    ],\n",
    "    correct=1, section=\"10. Narrowing\", bloom=\"understand\",\n",
    "    explanation=\"Narrowing prunes bad values based on evidence. The dimension stays but with fewer options. A minimum is preserved to prevent overfitting.\")"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Key Insight:** Narrowing is \"learning\" — the machine prunes bad options based on evidence. This makes subsequent rungs faster and more focused.\n",
    "\n",
    "---\n",
    "## 11. Cross-Rung Propagation\n",
    "\n",
    "A full **ratchet** chains multiple rungs. The winner from rung $N$ becomes the bootstrap for rung $N+1$."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "store3 = ResearchStore(tempfile.mkdtemp())\n",
    "harness3 = AutoresearchHarness(store3)\n",
    "\n",
    "rung1 = RungConfig(\n",
    "    rung=1, name=\"Rung 1\", description=\"Explore basics\",\n",
    "    objective=\"Find best seed and verification\",\n",
    "    bootstrap_incumbent=ExperimentSpec(\n",
    "        rung=1, target_backend=\"fake_brisbane\", noise_backend=\"fake_brisbane\",\n",
    "        shots=256, repeats=1,\n",
    "    ),\n",
    "    search_space=SearchSpaceConfig(\n",
    "        dimensions={\"verification\": [\"both\", \"z_only\"], \"seed_style\": [\"h_p\", \"ry_rz\"]},\n",
    "        max_challengers_per_step=4,\n",
    "    ),\n",
    "    tier_policy=TierPolicyConfig(cheap_margin=0.0, cheap_shots=256, cheap_repeats=1,\n",
    "                                  promote_top_k=1, enable_hardware=False),\n",
    "    score=rung_config.score, step_budget=2, patience=1, hardware=HardwareConfig(),\n",
    ")\n",
    "\n",
    "rung2 = RungConfig(\n",
    "    rung=2, name=\"Rung 2\", description=\"Refine optimization\",\n",
    "    objective=\"Tune optimization level\",\n",
    "    bootstrap_incumbent=ExperimentSpec(\n",
    "        rung=2, target_backend=\"fake_brisbane\", noise_backend=\"fake_brisbane\",\n",
    "        shots=256, repeats=1,\n",
    "    ),\n",
    "    search_space=SearchSpaceConfig(\n",
    "        dimensions={\"optimization_level\": [1, 2, 3], \"verification\": [\"both\", \"z_only\"]},\n",
    "        max_challengers_per_step=4,\n",
    "    ),\n",
    "    tier_policy=rung1.tier_policy,\n",
    "    score=rung_config.score, step_budget=2, patience=1, hardware=HardwareConfig(),\n",
    ")\n",
    "\n",
    "results = harness3.run_ratchet([rung1, rung2], allow_hardware=False)\n",
    "\n",
    "for lesson_obj, fb in results:\n",
    "    print(f\"\\nRung {lesson_obj.rung}: {lesson_obj.name}\")\n",
    "    print(f\"  Rules: {len(fb.rules)}\")\n",
    "    best_fields = {k: v for k, v in list(fb.best_spec_fields.items())[:4]}\n",
    "    print(f\"  Best spec: {best_fields}...\")\n",
    "\n",
    "print(f\"\\nAccumulated lessons: {len(harness3._accumulated_lessons)} rungs of feedback\")"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## 12. Transfer Evaluation\n",
    "\n",
    "A transfer test checks if the best settings generalize across different backend noise profiles (not overfit to one)."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "evaluator = TransferEvaluator()\n",
    "report = evaluator.evaluate_across_backends(\n",
    "    incumbent,\n",
    "    [\"fake_brisbane\"],  # Use single backend for speed\n",
    "    fast_rung,\n",
    ")\n",
    "print(f\"Transfer score (pessimistic = min): {report.transfer_score:.4f}\")\n",
    "print(f\"Mean score:  {report.mean_score:.4f}\")\n",
    "for name, score in report.per_backend_scores.items():\n",
    "    print(f\"  {name}: {score:.4f}\")"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "quiz(tracker, \"q10_transfer\",\n",
    "    question=\"A spec scores 0.8 on one backend but 0.3 on another. What does this mean?\",\n",
    "    options=[\n",
    "        \"The spec is bad overall\",\n",
    "        \"The spec is overfitted to the first backend's noise profile\",\n",
    "        \"The second backend is broken\",\n",
    "    ],\n",
    "    correct=1, section=\"12. Transfer\", bloom=\"evaluate\",\n",
    "    explanation=\"A large transfer drop means settings are tuned to one backend's quirks. The ratchet tests transfer to find robust, generalizable configurations.\")\n",
    "checkpoint_summary(tracker, \"12. Transfer\")"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Summary\n",
    "\n",
    "| Concept | What it does |\n",
    "|---|---|\n",
    "| **NeighborWalk** | Single-axis mutations — systematic but limited |\n",
    "| **RandomCombo** | Multi-axis mutations — discovers interactions |\n",
    "| **LessonGuided** | Rule-biased mutations — focuses on promising regions |\n",
    "| **Ratchet step** | Generate, evaluate, promote, select winner |\n",
    "| **Patience** | Stop early if no improvement |\n",
    "| **Lesson extraction** | Human-readable + machine-readable rules from data |\n",
    "| **Search narrowing** | Prune bad values, lock good ones |\n",
    "| **Cross-rung propagation** | Winner and lessons flow to the next rung |\n",
    "| **Transfer evaluation** | Check generalization across noise profiles |\n",
    "\n",
    "> **Dashboard Exercise:** Try to manually find the best configuration in `00_dashboard.ipynb`. Then compare your best score to what the ratchet found. Who wins?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Final Assessment"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "tracker.dashboard()\n",
    "path = tracker.save()\n",
    "print(f\"\\nProgress saved to: {path}\")"
   ],
   "outputs": [],
   "execution_count": null
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.14.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}