{ "cells": [ { "cell_type": "markdown", "id": "751fe8cc", "metadata": {}, "source": [ "# Notebook 3: The Ratchet Learns For You\n", "\n", "**Plan A — Automated Search**\n", "\n", "You now know what an encoded magic state is (Notebook 1) and how to measure its quality (Notebook 2). This notebook shows how the **autoresearch ratchet** automatically explores the parameter space to find the best circuit configuration.\n", "\n", "**What you will learn:**\n", "1. The incumbent-challenger optimization model\n", "2. How challengers are generated (neighbor walk, random combo, lesson-guided)\n", "3. How the ratchet selects winners and extracts lessons\n", "4. Cross-rung propagation and search space narrowing" ] }, { "cell_type": "code", "execution_count": 1, "id": "3f9b56a6", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "All imports successful.\n" ] } ], "source": [ "%matplotlib inline\n", "import sys, warnings, tempfile\n", "warnings.filterwarnings(\"ignore\")\n", "\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "from math import sqrt\n", "\n", "from autoresearch_quantum.models import (\n", " ExperimentSpec, RungConfig, EvaluationMetrics,\n", " QualityWeights, CostWeights, ScoreConfig, SearchSpaceConfig,\n", " TierPolicyConfig, HardwareConfig, LessonFeedback, SearchRule,\n", ")\n", "from autoresearch_quantum.execution.local import LocalCheapExecutor\n", "from autoresearch_quantum.search.challengers import (\n", " generate_neighbor_challengers, mutation_summary, GeneratedChallenger,\n", ")\n", "from autoresearch_quantum.search.strategies import (\n", " NeighborWalk, RandomCombo, LessonGuided, CompositeGenerator,\n", " default_composite, StrategyWeight,\n", ")\n", "from autoresearch_quantum.ratchet.runner import AutoresearchHarness\n", "from autoresearch_quantum.persistence.store import ResearchStore\n", "from autoresearch_quantum.config import load_rung_config\n", "from autoresearch_quantum.lessons.extractor import extract_rung_lesson\n", "from autoresearch_quantum.lessons.feedback import (\n", " extract_search_rules, narrow_search_space, build_lesson_feedback,\n", ")\n", "from autoresearch_quantum.execution.transfer import TransferEvaluator\n", "\n", "print(\"All imports successful.\")" ] }, { "cell_type": "code", "execution_count": 2, "id": "7cb035ce", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Learning tracker active.\n" ] } ], "source": [ "from autoresearch_quantum.teaching import LearningTracker\n", "from autoresearch_quantum.teaching.assess import quiz, predict_choice, reflect, order, checkpoint_summary\n", "tracker = LearningTracker(\"plan_a_03\")\n", "print(\"Learning tracker active.\")" ] }, { "cell_type": "markdown", "id": "5fefc4e4", "metadata": {}, "source": [ "---\n", "## 1. The Incumbent-Challenger Model\n", "\n", "The ratchet keeps a **best-so-far** configuration called the **incumbent**. Each step:\n", "\n", "1. Generate **challengers** — new configurations that differ from the incumbent in one or more parameters\n", "2. Evaluate each challenger on the cheap tier (noisy simulator)\n", "3. If any challenger beats the incumbent by a margin, it becomes the new incumbent\n", "4. Repeat until patience runs out\n", "\n", "This is a form of **local search** — like hill climbing in parameter space." ] }, { "cell_type": "code", "execution_count": 3, "id": "e563c118", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Bootstrap incumbent:\n", " seed_style: h_p\n", " encoder_style: cx_chain\n", " verification: both\n", " postselection: all_measured\n", " optimization_level: 2\n", " target_backend: fake_brisbane\n", "\n", "Search space dimensions:\n", " seed_style: ['h_p', 'ry_rz', 'u_magic']\n", " encoder_style: ['cx_chain', 'cz_compiled']\n", " verification: ['both', 'z_only', 'x_only']\n", " postselection: ['all_measured', 'z_only', 'none']\n", " ancilla_strategy: ['dedicated_pair', 'reused_single']\n", " optimization_level: [1, 2, 3]\n", "\n", "Max challengers per step: 8\n" ] } ], "source": [ "# Load the rung1 configuration\n", "rung_config = load_rung_config(\"../../configs/rungs/rung1.yaml\")\n", "\n", "# The bootstrap incumbent\n", "incumbent_spec = rung_config.bootstrap_incumbent\n", "print(\"Bootstrap incumbent:\")\n", "print(f\" seed_style: {incumbent_spec.seed_style}\")\n", "print(f\" encoder_style: {incumbent_spec.encoder_style}\")\n", "print(f\" verification: {incumbent_spec.verification}\")\n", "print(f\" postselection: {incumbent_spec.postselection}\")\n", "print(f\" optimization_level: {incumbent_spec.optimization_level}\")\n", "print(f\" target_backend: {incumbent_spec.target_backend}\")\n", "print(f\"\\nSearch space dimensions:\")\n", "for dim, values in rung_config.search_space.dimensions.items():\n", " print(f\" {dim}: {values}\")\n", "print(f\"\\nMax challengers per step: {rung_config.search_space.max_challengers_per_step}\")" ] }, { "cell_type": "markdown", "id": "d4044fc8", "metadata": {}, "source": [ "### The ratchet guarantee\n", "\n", "The key property: the incumbent **never gets worse**. A challenger must demonstrably beat the incumbent to replace it." ] }, { "cell_type": "code", "execution_count": 4, "id": "1f48aa77", "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "47c3d5bbac3d4fd4ab7c8c57b4432b17", "version_major": 2, "version_minor": 0 }, "text/plain": [ "VBox(children=(HTML(value='
…" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "47c3d5bbac3d4fd4ab7c8c57b4432b17", "version_major": 2, "version_minor": 0 }, "text/plain": [ "VBox(children=(HTML(value='
…" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "quiz(tracker, \"q1_ratchet_guarantee\",\n", " question=\"What is the ratchet guarantee?\",\n", " options=[\n", " \"Every step improves the score\",\n", " \"The incumbent never gets worse \\u2014 challengers must beat it to replace it\",\n", " \"The search space shrinks every step\",\n", " \"The ratchet always converges to the global optimum\",\n", " ],\n", " correct=1, section=\"1. Incumbent-challenger\", bloom=\"remember\",\n", " explanation=\"The ratchet is monotonic: if no challenger beats the incumbent, the incumbent stays. This does NOT guarantee finding the global optimum.\")" ] }, { "cell_type": "markdown", "id": "07aea2c1", "metadata": {}, "source": [ "---\n", "## 2. Generating Challengers: Neighbor Walk\n", "\n", "The simplest strategy: change **one parameter at a time**. For each dimension in the search space, try each alternative value." ] }, { "cell_type": "code", "execution_count": null, "id": "9bb9a7f8", "metadata": {}, "outputs": [], "source": [ "challengers = generate_neighbor_challengers(\n", " incumbent_spec,\n", " rung_config.search_space,\n", ")\n", "\n", "print(f\"Generated {len(challengers)} challengers:\\n\")\n", "for i, c in enumerate(challengers):\n", " print(f\" {i+1}. {c.mutation_note}\")" ] }, { "cell_type": "code", "execution_count": null, "id": "1d3c7add", "metadata": {}, "outputs": [], "source": [ "quiz(tracker, \"q2_neighborwalk\",\n", " question=\"How does NeighborWalk generate challengers?\",\n", " options=[\n", " \"Changes all parameters simultaneously to random values\",\n", " \"Changes exactly one parameter at a time to each of its other possible values\",\n", " \"Applies gradient descent to continuous parameters\",\n", " ],\n", " correct=1, section=\"2. Challengers\", bloom=\"understand\",\n", " explanation=\"NeighborWalk is single-axis: for each dimension, try every alternative value while keeping all other dimensions fixed.\")\n", "checkpoint_summary(tracker, \"2. Challengers\")" ] }, { "cell_type": "markdown", "id": "42943480", "metadata": {}, "source": [ "> **Key Insight:** Neighbor walk is exhaustive within one axis but never explores *combinations* of changes. It is good for identifying which single parameter matters most." ] }, { "cell_type": "markdown", "id": "ed8e7c15", "metadata": {}, "source": [ "---\n", "## 3. Evaluating Challengers\n", "\n", "Let us run the incumbent and all challengers on the local noisy simulator." ] }, { "cell_type": "code", "execution_count": null, "id": "369bf954", "metadata": {}, "outputs": [], "source": [ "# Use smaller settings for speed\n", "fast_rung = RungConfig(\n", " rung=1, name=rung_config.name, description=rung_config.description,\n", " objective=rung_config.objective, bootstrap_incumbent=incumbent_spec,\n", " search_space=rung_config.search_space,\n", " tier_policy=TierPolicyConfig(\n", " cheap_margin=0.002, cheap_shots=256, cheap_repeats=1,\n", " expensive_shots=512, expensive_repeats=1,\n", " promote_top_k=2, enable_hardware=False,\n", " ),\n", " score=rung_config.score,\n", " step_budget=1, patience=1,\n", " hardware=HardwareConfig(),\n", ")\n", "\n", "executor = LocalCheapExecutor()\n", "incumbent_result = executor.evaluate(incumbent_spec, fast_rung)\n", "print(f\"Incumbent score: {incumbent_result.score:.4f}\")\n", "print(f\" failure_mode: {incumbent_result.metrics.dominant_failure_mode}\")\n", "\n", "challenger_scores = {}\n", "for c in challengers[:8]: # Limit for speed\n", " result = executor.evaluate(c.spec, fast_rung)\n", " challenger_scores[c.mutation_note] = result.score\n", " beat = \"BEATS\" if result.score > incumbent_result.score else \" \"\n", " print(f\" {beat} {c.mutation_note}: {result.score:.4f}\")" ] }, { "cell_type": "code", "execution_count": null, "id": "3d80e869", "metadata": {}, "outputs": [], "source": [ "# Visualize incumbent vs challengers\n", "fig, ax = plt.subplots(figsize=(12, 5))\n", "\n", "labels = [\"INCUMBENT\"] + list(challenger_scores.keys())\n", "scores = [incumbent_result.score] + list(challenger_scores.values())\n", "colors = [\"#e74c3c\"] + [\"#2ecc71\" if s > incumbent_result.score else \"#95a5a6\" for s in challenger_scores.values()]\n", "\n", "bars = ax.barh(range(len(labels)), scores, color=colors)\n", "ax.set_yticks(range(len(labels)))\n", "ax.set_yticklabels([l[:40] for l in labels], fontsize=8)\n", "ax.set_xlabel(\"Score\")\n", "ax.set_title(\"Incumbent vs Challengers\")\n", "ax.axvline(x=incumbent_result.score, color=\"#e74c3c\", linestyle=\"--\", alpha=0.5, label=\"Incumbent\")\n", "ax.legend()\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": null, "id": "8d43f418", "metadata": {}, "outputs": [], "source": [ "predict_choice(tracker, \"q3_challenger_wins\",\n", " question=\"Looking at the bar chart: did any challenger beat the incumbent?\",\n", " options=[\n", " \"Yes \\u2014 at least one bar is taller than INCUMBENT\",\n", " \"No \\u2014 the incumbent bar is the tallest\",\n", " \"Can't tell from a bar chart\",\n", " ],\n", " correct=0, section=\"3. Evaluation\", bloom=\"apply\",\n", " explanation=\"In most runs, at least one challenger finds a better configuration.\")" ] }, { "cell_type": "markdown", "id": "9da23164", "metadata": {}, "source": [ "---\n", "## 4. One Ratchet Step in Slow Motion\n", "\n", "Now let the harness run a complete ratchet step — including challenger generation, evaluation, promotion, and winner selection." ] }, { "cell_type": "code", "execution_count": null, "id": "788ec6fd", "metadata": {}, "outputs": [], "source": [ "store = ResearchStore(tempfile.mkdtemp())\n", "harness = AutoresearchHarness(store)\n", "\n", "step = harness.run_ratchet_step(fast_rung, allow_hardware=False)\n", "\n", "print(f\"Step index: {step.step_index}\")\n", "print(f\"Incumbent before: {step.incumbent_before_id}\")\n", "print(f\"Challengers tested: {len(step.challengers_tested)}\")\n", "print(f\"Promoted to expensive: {len(step.promoted_challengers)}\")\n", "print(f\"Winner: {step.winner_id}\")\n", "print(f\"Winning margin: {step.winning_margin:+.4f}\")\n", "print(f\"\\nCheap-tier justification:\")\n", "print(f\" {step.cheap_tier_justification}\")\n", "print(f\"\\nDistilled lesson:\")\n", "print(f\" {step.distilled_lesson}\")" ] }, { "cell_type": "code", "execution_count": null, "id": "979057fb", "metadata": {}, "outputs": [], "source": [ "quiz(tracker, \"q4_no_improvement\",\n", " question=\"What happens if ALL challengers score lower than the incumbent?\",\n", " options=[\n", " \"The harness picks the best challenger anyway\",\n", " \"The incumbent stays and the step is logged with zero improvement\",\n", " \"The harness generates more challengers until one wins\",\n", " ],\n", " correct=1, section=\"4. Ratchet step\", bloom=\"understand\",\n", " explanation=\"Monotonic guarantee: if no challenger wins, the incumbent stays. Consecutive no-improvement steps trigger patience.\")\n", "checkpoint_summary(tracker, \"4. Ratchet step\")" ] }, { "cell_type": "markdown", "id": "bfb10c0e", "metadata": {}, "source": [ "> **Key Insight:** The tells you how much the winner improved over the incumbent. If positive, the ratchet \"clicked\" forward. The is a human-readable summary of what changed and why.\n", "\n", "---\n", "## 5. Running a Full Rung\n", "\n", "A **rung** runs multiple ratchet steps in sequence. It stops when the step budget is exhausted or when patience runs out (no improvement for N consecutive steps)." ] }, { "cell_type": "code", "execution_count": null, "id": "a7123aa0", "metadata": {}, "outputs": [], "source": [ "# Fresh store for a clean rung\n", "store2 = ResearchStore(tempfile.mkdtemp())\n", "harness2 = AutoresearchHarness(store2)\n", "\n", "# Run with step_budget=3\n", "full_rung = RungConfig(\n", " rung=1, name=rung_config.name, description=rung_config.description,\n", " objective=rung_config.objective, bootstrap_incumbent=incumbent_spec,\n", " search_space=SearchSpaceConfig(\n", " dimensions={\"verification\": [\"both\", \"z_only\", \"x_only\"],\n", " \"seed_style\": [\"h_p\", \"ry_rz\", \"u_magic\"],\n", " \"postselection\": [\"all_measured\", \"z_only\", \"none\"]},\n", " max_challengers_per_step=6,\n", " ),\n", " tier_policy=TierPolicyConfig(\n", " cheap_margin=0.001, cheap_shots=256, cheap_repeats=1,\n", " expensive_shots=512, expensive_repeats=1,\n", " promote_top_k=2, enable_hardware=False,\n", " ),\n", " score=rung_config.score,\n", " step_budget=3, patience=2,\n", " hardware=HardwareConfig(),\n", ")\n", "\n", "steps, lesson, feedback = harness2.run_rung(full_rung, allow_hardware=False)\n", "print(f\"Steps completed: {len(steps)}\")\n", "for s in steps:\n", " print(f\" Step {s.step_index}: winner={s.winner_id[:30]}... margin={s.winning_margin:+.4f}\")" ] }, { "cell_type": "code", "execution_count": null, "id": "bdeac1f5", "metadata": {}, "outputs": [], "source": [ "# Show the lesson\n", "print(\"=\" * 60)\n", "print(lesson.narrative)\n", "print(\"=\" * 60)" ] }, { "cell_type": "code", "execution_count": null, "id": "5ed4a4f2", "metadata": {}, "outputs": [], "source": [ "reflect(tracker, \"q5_lesson_quality\",\n", " question=\"Read the lesson narrative above. What actionable insight does it give? What would make it better?\",\n", " section=\"5. Lesson\", bloom=\"evaluate\",\n", " model_answer=\"A good lesson names specific parameter values that helped/hurt and explains WHY. The machine-readable rules are often more actionable than the narrative.\")" ] }, { "cell_type": "markdown", "id": "c016ebc3", "metadata": {}, "source": [ "---\n", "## 6. Visualizing the Exploration\n", "\n", "Let us plot how the score evolved and which experiments the ratchet tried." ] }, { "cell_type": "code", "execution_count": null, "id": "853caf26", "metadata": {}, "outputs": [], "source": [ "experiments = store2.list_experiments(1)\n", "exp_scores = [(e[\"experiment_id\"][:25], e[\"final_score\"], e[\"role\"]) for e in experiments]\n", "\n", "fig, ax = plt.subplots(figsize=(12, 5))\n", "colors = [\"#e74c3c\" if role == \"incumbent\" else \"#3498db\" for _, _, role in exp_scores]\n", "ax.bar(range(len(exp_scores)), [s for _, s, _ in exp_scores], color=colors)\n", "ax.set_xticks(range(len(exp_scores)))\n", "ax.set_xticklabels([eid for eid, _, _ in exp_scores], rotation=45, ha=\"right\", fontsize=7)\n", "ax.set_ylabel(\"Score\")\n", "ax.set_title(\"All Experiments in Rung 1\")\n", "\n", "# Add legend\n", "from matplotlib.patches import Patch\n", "ax.legend(handles=[Patch(color=\"#e74c3c\", label=\"Incumbent\"), Patch(color=\"#3498db\", label=\"Challenger\")])\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "77264cbd", "metadata": {}, "source": [ "---\n", "## 7. Search Strategies Compared\n", "\n", "The harness supports three strategies. Let us see how they generate different challengers from the same incumbent." ] }, { "cell_type": "code", "execution_count": null, "id": "d3871da7", "metadata": {}, "outputs": [], "source": [ "search_space = SearchSpaceConfig(\n", " dimensions={\n", " \"verification\": [\"both\", \"z_only\", \"x_only\"],\n", " \"seed_style\": [\"h_p\", \"ry_rz\", \"u_magic\"],\n", " \"optimization_level\": [1, 2, 3],\n", " },\n", " max_challengers_per_step=6,\n", ")\n", "\n", "for name, strategy in [(\"NeighborWalk\", NeighborWalk()),\n", " (\"RandomCombo\", RandomCombo(num_candidates=6)),\n", " (\"CompositeGenerator\", default_composite(has_lessons=False))]:\n", " challengers = strategy.generate(incumbent_spec, search_space, set())\n", " print(f\"\\n{name} ({len(challengers)} challengers):\")\n", " for c in challengers[:4]:\n", " print(f\" {c.mutation_note}\")\n", " if len(challengers) > 4:\n", " print(f\" ... and {len(challengers) - 4} more\")" ] }, { "cell_type": "markdown", "id": "b2b6b18f", "metadata": {}, "source": [ "### Strategy comparison\n", "\n", "- **NeighborWalk**: 1 axis at a time, systematic\n", "- **RandomCombo**: multiple axes, random\n", "- **LessonGuided**: rule-biased from previous rungs" ] }, { "cell_type": "code", "execution_count": null, "id": "fc0d6390", "metadata": {}, "outputs": [], "source": [ "order(tracker, \"q6_strategy_breadth\",\n", " instruction=\"Rank strategies from narrowest to broadest exploration:\",\n", " items=[\"NeighborWalk\", \"RandomCombo\", \"LessonGuided\"],\n", " correct_order=[\"NeighborWalk\", \"LessonGuided\", \"RandomCombo\"],\n", " section=\"6. Search strategies\", bloom=\"analyze\",\n", " explanation=\"NeighborWalk: 1 param (narrowest). LessonGuided: focused by rules (medium). RandomCombo: multiple params randomly (broadest).\")" ] }, { "cell_type": "markdown", "id": "1fb3757c", "metadata": {}, "source": [ "---\n", "## 8. Lesson-Guided Search\n", "\n", "After a rung completes, the harness extracts **SearchRules** — machine-readable directives like \"prefer z_only\" or \"avoid x_only\". These guide future search." ] }, { "cell_type": "code", "execution_count": null, "id": "d37dcb46", "metadata": {}, "outputs": [], "source": [ "# Show the feedback rules from our rung\n", "print(f\"Feedback from rung 1: {len(feedback.rules)} rules\\n\")\n", "for rule in feedback.rules:\n", " print(f\" {rule.action.upper():7s} {rule.dimension}={rule.value} (confidence={rule.confidence:.2f})\")\n", " print(f\" {rule.reason}\\n\")\n", "\n", "print(f\"Narrowed dimensions:\")\n", "for dim, vals in feedback.narrowed_dimensions.items():\n", " print(f\" {dim}: {vals}\")" ] }, { "cell_type": "code", "execution_count": null, "id": "2d485ab3", "metadata": {}, "outputs": [], "source": [ "quiz(tracker, \"q7_fix_vs_avoid\",\n", " question=\"What is the difference between a 'fix' rule and an 'avoid' rule?\",\n", " options=[\n", " \"'fix' locks a value permanently; 'avoid' removes a value from the search space\",\n", " \"'fix' repairs a bug; 'avoid' prevents a crash\",\n", " \"They are synonyms\",\n", " ],\n", " correct=0, section=\"7. Lesson-guided\", bloom=\"remember\",\n", " explanation=\"'fix': this value is clearly best, always use it. 'avoid': this value consistently hurts, remove it.\")\n", "checkpoint_summary(tracker, \"7. Lesson-guided\")" ] }, { "cell_type": "code", "execution_count": null, "id": "09951547", "metadata": {}, "outputs": [], "source": [ "# Run LessonGuided strategy using the feedback\n", "if feedback.rules:\n", " guided = LessonGuided(num_candidates=6)\n", " guided_challengers = guided.generate(incumbent_spec, search_space, set(), [feedback])\n", " print(f\"Lesson-guided generated {len(guided_challengers)} challengers:\")\n", " for c in guided_challengers:\n", " print(f\" {c.mutation_note}\")\n", "else:\n", " print(\"No rules extracted (too few experiments). Try increasing step_budget.\")" ] }, { "cell_type": "markdown", "id": "24e1dcfb", "metadata": {}, "source": [ "---\n", "## 9. Cross-Rung Propagation\n", "\n", "A **ratchet** runs multiple rungs in sequence. The winner from rung N becomes the bootstrap incumbent for rung N+1. Lessons narrow the search space." ] }, { "cell_type": "code", "execution_count": null, "id": "6730be92", "metadata": {}, "outputs": [], "source": [ "store3 = ResearchStore(tempfile.mkdtemp())\n", "harness3 = AutoresearchHarness(store3)\n", "\n", "rung1_config = RungConfig(\n", " rung=1, name=\"Rung 1\", description=\"Explore seed and verification\",\n", " objective=\"Find best basic config\",\n", " bootstrap_incumbent=ExperimentSpec(\n", " rung=1, target_backend=\"fake_brisbane\", noise_backend=\"fake_brisbane\",\n", " shots=256, repeats=1,\n", " ),\n", " search_space=SearchSpaceConfig(\n", " dimensions={\"verification\": [\"both\", \"z_only\"], \"seed_style\": [\"h_p\", \"ry_rz\"]},\n", " max_challengers_per_step=4,\n", " ),\n", " tier_policy=TierPolicyConfig(cheap_margin=0.0, cheap_shots=256, cheap_repeats=1,\n", " promote_top_k=1, enable_hardware=False),\n", " score=rung_config.score, step_budget=2, patience=1, hardware=HardwareConfig(),\n", ")\n", "\n", "rung2_config = RungConfig(\n", " rung=2, name=\"Rung 2\", description=\"Refine with more dimensions\",\n", " objective=\"Optimize further\",\n", " bootstrap_incumbent=ExperimentSpec(\n", " rung=2, target_backend=\"fake_brisbane\", noise_backend=\"fake_brisbane\",\n", " shots=256, repeats=1,\n", " ),\n", " search_space=SearchSpaceConfig(\n", " dimensions={\"verification\": [\"both\", \"z_only\"], \"optimization_level\": [1, 2, 3]},\n", " max_challengers_per_step=4,\n", " ),\n", " tier_policy=rung1_config.tier_policy,\n", " score=rung_config.score, step_budget=2, patience=1, hardware=HardwareConfig(),\n", ")\n", "\n", "results = harness3.run_ratchet([rung1_config, rung2_config], allow_hardware=False)\n", "\n", "for lesson_obj, fb in results:\n", " print(f\"\\nRung {lesson_obj.rung}: {lesson_obj.name}\")\n", " print(f\" Rules extracted: {len(fb.rules)}\")\n", " print(f\" Best spec fields: {dict(list(fb.best_spec_fields.items())[:5])}...\")" ] }, { "cell_type": "code", "execution_count": null, "id": "3da85dc8", "metadata": {}, "outputs": [], "source": [ "quiz(tracker, \"q8_propagation\",\n", " question=\"Why does the ratchet propagate the winning spec to the next rung?\",\n", " options=[\n", " \"To save typing the spec again\",\n", " \"The winner from rung N is a good starting point for rung N+1, avoiding cold-start\",\n", " \"Each rung must use the same spec\",\n", " ],\n", " correct=1, section=\"8. Cross-rung\", bloom=\"understand\",\n", " explanation=\"Cross-rung propagation transfers knowledge: best settings from one rung become the starting point for the next.\")" ] }, { "cell_type": "markdown", "id": "59c7bef1", "metadata": {}, "source": [ "---\n", "## 10. Transfer Evaluation\n", "\n", "A **transfer test** runs the best spec across multiple backend noise models to check if the settings generalize or are overfit to one specific noise profile." ] }, { "cell_type": "code", "execution_count": null, "id": "5628cccb", "metadata": {}, "outputs": [], "source": [ "evaluator = TransferEvaluator()\n", "report = evaluator.evaluate_across_backends(\n", " incumbent_spec,\n", " [\"fake_brisbane\"], # Single backend for speed (add more for real transfer tests)\n", " fast_rung,\n", ")\n", "\n", "print(f\"Transfer score (pessimistic = min): {report.transfer_score:.4f}\")\n", "print(f\"Mean score: {report.mean_score:.4f}\")\n", "print(f\"Std score: {report.std_score:.4f}\")\n", "for backend_name, score in report.per_backend_scores.items():\n", " print(f\" {backend_name}: {score:.4f}\")" ] }, { "cell_type": "code", "execution_count": null, "id": "5dea8979", "metadata": {}, "outputs": [], "source": [ "quiz(tracker, \"q9_transfer_quality\",\n", " question=\"When is a transfer score 'good'?\",\n", " options=[\n", " \"When it is higher than 0\",\n", " \"When it is close to the original score on the source backend\",\n", " \"When it is exactly 1.0\",\n", " ],\n", " correct=1, section=\"9. Transfer\", bloom=\"evaluate\",\n", " explanation=\"Good transfer means settings work almost as well on the target backend. A large drop means overfitting to the source noise profile.\")\n", "checkpoint_summary(tracker, \"9. Transfer\")" ] }, { "cell_type": "markdown", "id": "71095cd5", "metadata": {}, "source": [ "---\n", "## Summary\n", "\n", "You have now seen the complete autoresearch pipeline:\n", "\n", "| Layer | What happens |\n", "|---|---|\n", "| **Circuit** | Build encoded magic state (Notebook 1) |\n", "| **Metrics** | Measure quality, cost, and score (Notebook 2) |\n", "| **Search** | Generate and evaluate challengers (this notebook) |\n", "| **Ratchet** | Iterate: incumbent vs challengers, promote winners |\n", "| **Lessons** | Extract rules, narrow search space, propagate to next rung |\n", "| **Transfer** | Verify settings generalize across backends |\n", "\n", "The entire process compresses hours of manual parameter exploration into minutes of automated search. Each rung produces a human-readable lesson and machine-readable rules that make future exploration more efficient." ] }, { "cell_type": "markdown", "id": "ba79fac4", "metadata": {}, "source": [ "---\n", "## Final Assessment" ] }, { "cell_type": "code", "execution_count": null, "id": "bdbf806a", "metadata": {}, "outputs": [], "source": [ "tracker.dashboard()\n", "path = tracker.save()\n", "print(f\"\\nProgress saved to: {path}\")" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.14.2" } }, "nbformat": 4, "nbformat_minor": 5 }