mirror of
https://github.com/saymrwulf/autoresearch-quantum.git
synced 2026-09-04 18:53:39 +00:00
saymrwulf
18f5bef127
- Create notebooks/00_START_HERE.ipynb as the single entry point with plan descriptions, audience guidance, and links to all 4 plans - Add navigation footer cells to all 11 content notebooks with Next/Previous links and back-link to Start Here - Terminal notebooks (plan endings) offer cross-plan links to explore other plans - Plan C dashboard gets explicit recommended reading order (Track A → B → C) - Add test_start_here_exists_and_links_all_plans and test_every_notebook_has_navigation_footer to test suite - Skip navigation-only notebooks in code-cell and assessment tests
761 lines
No EOL
28 KiB
Text
761 lines
No EOL
28 KiB
Text
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Track C: The Search \u2014 Optimization and the Ratchet\n",
|
|
"\n",
|
|
"**Plan C \u2014 Parallel Tracks**\n",
|
|
"\n",
|
|
"This track covers automated parameter search. You will learn how the ratchet generates challengers, selects winners, extracts lessons, and narrows the search space across rungs.\n",
|
|
"\n",
|
|
"> **Dashboard:** Use `00_dashboard.ipynb` to explore individual experiments manually, then see how the ratchet does it automatically."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"metadata": {},
|
|
"source": [
|
|
"%matplotlib inline\n",
|
|
"import warnings, tempfile\n",
|
|
"warnings.filterwarnings(\"ignore\")\n",
|
|
"\n",
|
|
"import numpy as np\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"from math import sqrt\n",
|
|
"\n",
|
|
"from autoresearch_quantum.models import (\n",
|
|
" ExperimentSpec, RungConfig, EvaluationMetrics,\n",
|
|
" QualityWeights, CostWeights, ScoreConfig, SearchSpaceConfig,\n",
|
|
" TierPolicyConfig, HardwareConfig, LessonFeedback, SearchRule,\n",
|
|
")\n",
|
|
"from autoresearch_quantum.execution.local import LocalCheapExecutor\n",
|
|
"from autoresearch_quantum.search.challengers import (\n",
|
|
" generate_neighbor_challengers, mutation_summary, GeneratedChallenger,\n",
|
|
")\n",
|
|
"from autoresearch_quantum.search.strategies import (\n",
|
|
" NeighborWalk, RandomCombo, LessonGuided, CompositeGenerator,\n",
|
|
" default_composite, StrategyWeight,\n",
|
|
")\n",
|
|
"from autoresearch_quantum.ratchet.runner import AutoresearchHarness\n",
|
|
"from autoresearch_quantum.persistence.store import ResearchStore\n",
|
|
"from autoresearch_quantum.config import load_rung_config\n",
|
|
"from autoresearch_quantum.lessons.extractor import extract_rung_lesson\n",
|
|
"from autoresearch_quantum.lessons.feedback import (\n",
|
|
" extract_search_rules, narrow_search_space, build_lesson_feedback,\n",
|
|
")\n",
|
|
"from autoresearch_quantum.execution.transfer import TransferEvaluator\n",
|
|
"from matplotlib.patches import Patch\n",
|
|
"\n",
|
|
"print(\"All imports successful.\")"
|
|
],
|
|
"outputs": [],
|
|
"execution_count": null
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"metadata": {},
|
|
"source": [
|
|
"from autoresearch_quantum.teaching import LearningTracker\n",
|
|
"from autoresearch_quantum.teaching.assess import quiz, predict_choice, reflect, order, checkpoint_summary\n",
|
|
"tracker = LearningTracker(\"plan_c_track_c\")\n",
|
|
"print(\"Learning tracker active.\")"
|
|
],
|
|
"outputs": [],
|
|
"execution_count": null
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"---\n",
|
|
"## 1. The Parameter Space\n",
|
|
"\n",
|
|
"The rung1 config defines a discrete search space over circuit parameters."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"metadata": {},
|
|
"source": [
|
|
"rung_config = load_rung_config(\"../../configs/rungs/rung1.yaml\")\n",
|
|
"\n",
|
|
"print(f\"Rung: {rung_config.name}\")\n",
|
|
"print(f\"Objective: {rung_config.objective}\")\n",
|
|
"print(f\"\\nSearch dimensions:\")\n",
|
|
"total_combos = 1\n",
|
|
"for dim, values in rung_config.search_space.dimensions.items():\n",
|
|
" print(f\" {dim:25s}: {values}\")\n",
|
|
" total_combos *= len(values)\n",
|
|
"print(f\"\\nTotal combinations: {total_combos}\")\n",
|
|
"print(f\"Max challengers per step: {rung_config.search_space.max_challengers_per_step}\")\n",
|
|
"print(f\"Step budget: {rung_config.step_budget}\")\n",
|
|
"print(f\"Patience: {rung_config.patience}\")"
|
|
],
|
|
"outputs": [],
|
|
"execution_count": null
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"metadata": {},
|
|
"source": [
|
|
"quiz(tracker, \"q1_why_search\",\n",
|
|
" question=\"The parameter space is finite. Why not just try every combination?\",\n",
|
|
" options=[\n",
|
|
" \"The space is infinite\",\n",
|
|
" \"Each evaluation costs time/compute; smart search finds good solutions faster\",\n",
|
|
" \"Exhaustive search always finds worse solutions\",\n",
|
|
" ],\n",
|
|
" correct=1, section=\"1. Parameter space\", bloom=\"understand\",\n",
|
|
" explanation=\"Each evaluation requires noisy simulation (or hardware QPU time). Smart search finds good solutions in fewer evaluations.\")"
|
|
],
|
|
"outputs": [],
|
|
"execution_count": null
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"With {total} combinations, exhaustive search is feasible but slow. The ratchet is smarter: it starts from a good baseline and explores *neighborhoods*, focusing compute where it matters.\n",
|
|
"\n",
|
|
"---\n",
|
|
"## 2. The Incumbent-Challenger Model\n",
|
|
"\n",
|
|
"Think of it like a chess championship:\n",
|
|
"- The **incumbent** is the reigning champion (best configuration found so far)\n",
|
|
"- **Challengers** are generated by mutating the incumbent's parameters\n",
|
|
"- Each challenger is evaluated (\"plays a match\")\n",
|
|
"- If a challenger beats the incumbent by a sufficient margin, it takes the title\n",
|
|
"\n",
|
|
"This is a form of **local search** \u2014 like hill climbing in a discrete parameter space."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"metadata": {},
|
|
"source": [
|
|
"incumbent = rung_config.bootstrap_incumbent\n",
|
|
"print(\"Bootstrap incumbent:\")\n",
|
|
"for field in [\"seed_style\", \"encoder_style\", \"verification\", \"postselection\",\n",
|
|
" \"ancilla_strategy\", \"optimization_level\"]:\n",
|
|
" print(f\" {field:25s}: {getattr(incumbent, field)}\")"
|
|
],
|
|
"outputs": [],
|
|
"execution_count": null
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"metadata": {},
|
|
"source": [
|
|
"quiz(tracker, \"q2_incumbent\",\n",
|
|
" question=\"What is the bootstrap incumbent?\",\n",
|
|
" options=[\n",
|
|
" \"A randomly chosen starting point\",\n",
|
|
" \"A hand-picked reasonable default that the ratchet tries to beat\",\n",
|
|
" \"The theoretically optimal configuration\",\n",
|
|
" ],\n",
|
|
" correct=1, section=\"2. Incumbent\", bloom=\"remember\",\n",
|
|
" explanation=\"The bootstrap incumbent is a domain-expert guess. The ratchet guarantee: it never gets worse from here.\")\n",
|
|
"checkpoint_summary(tracker, \"2. Incumbent\")"
|
|
],
|
|
"outputs": [],
|
|
"execution_count": null
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"---\n",
|
|
"## 3. NeighborWalk: Single-Axis Perturbation\n",
|
|
"\n",
|
|
"The simplest strategy: change **one parameter at a time** and try all alternatives."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"metadata": {},
|
|
"source": [
|
|
"challengers = generate_neighbor_challengers(incumbent, rung_config.search_space)\n",
|
|
"\n",
|
|
"print(f\"Generated {len(challengers)} challengers (max {rung_config.search_space.max_challengers_per_step}):\\n\")\n",
|
|
"for i, c in enumerate(challengers):\n",
|
|
" print(f\" {i+1:2d}. {c.mutation_note}\")"
|
|
],
|
|
"outputs": [],
|
|
"execution_count": null
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"metadata": {},
|
|
"source": [
|
|
"quiz(tracker, \"q3_neighborwalk\",\n",
|
|
" question=\"NeighborWalk changes how many parameters per challenger?\",\n",
|
|
" options=[\"0\", \"Exactly 1\", \"Up to 3\", \"All of them\"],\n",
|
|
" correct=1, section=\"3. NeighborWalk\", bloom=\"understand\",\n",
|
|
" explanation=\"One parameter at a time, trying all alternative values. Systematic but blind to parameter interactions.\")"
|
|
],
|
|
"outputs": [],
|
|
"execution_count": null
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"> **Key Insight:** NeighborWalk is exhaustive within single axes but never tests *combinations*. It is fast and deterministic \u2014 good for identifying which individual parameter matters most.\n",
|
|
"\n",
|
|
"---\n",
|
|
"## 4. RandomCombo: Multi-Axis Perturbation\n",
|
|
"\n",
|
|
"Change 1-3 parameters simultaneously."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"metadata": {},
|
|
"source": [
|
|
"combo = RandomCombo(num_candidates=8, max_mutations=3)\n",
|
|
"combo_challengers = combo.generate(incumbent, rung_config.search_space, set())\n",
|
|
"\n",
|
|
"print(f\"Generated {len(combo_challengers)} random combo challengers:\\n\")\n",
|
|
"for i, c in enumerate(combo_challengers):\n",
|
|
" # Count how many fields changed\n",
|
|
" n_changes = sum(1 for f in incumbent.__dataclass_fields__\n",
|
|
" if getattr(incumbent, f) != getattr(c.spec, f))\n",
|
|
" print(f\" {i+1:2d}. [{n_changes} changes] {c.mutation_note}\")"
|
|
],
|
|
"outputs": [],
|
|
"execution_count": null
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"metadata": {},
|
|
"source": [
|
|
"order(tracker, \"q4_strategy_interactions\",\n",
|
|
" instruction=\"Rank strategies by ability to find multi-parameter interactions (worst to best):\",\n",
|
|
" items=[\"NeighborWalk\", \"RandomCombo\"],\n",
|
|
" correct_order=[\"NeighborWalk\", \"RandomCombo\"],\n",
|
|
" section=\"4. RandomCombo\", bloom=\"analyze\",\n",
|
|
" explanation=\"NeighborWalk: 1 axis only, cannot find interactions. RandomCombo mutates multiple axes simultaneously.\")\n",
|
|
"checkpoint_summary(tracker, \"4. RandomCombo\")"
|
|
],
|
|
"outputs": [],
|
|
"execution_count": null
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"> **Key Insight:** RandomCombo can discover *interaction effects* \u2014 parameter combinations that are better or worse than the sum of individual effects. It introduces diversity but is less systematic.\n",
|
|
"\n",
|
|
"---\n",
|
|
"## 5. Evaluating: Incumbent vs Challengers\n",
|
|
"\n",
|
|
"Let us actually run the experiments and see scores."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Use fast settings\n",
|
|
"fast_rung = RungConfig(\n",
|
|
" rung=1, name=rung_config.name, description=rung_config.description,\n",
|
|
" objective=rung_config.objective, bootstrap_incumbent=incumbent,\n",
|
|
" search_space=rung_config.search_space,\n",
|
|
" tier_policy=TierPolicyConfig(\n",
|
|
" cheap_margin=0.002, cheap_shots=256, cheap_repeats=1,\n",
|
|
" expensive_shots=512, expensive_repeats=1,\n",
|
|
" promote_top_k=2, enable_hardware=False,\n",
|
|
" ),\n",
|
|
" score=rung_config.score,\n",
|
|
" step_budget=1, patience=1, hardware=HardwareConfig(),\n",
|
|
")\n",
|
|
"\n",
|
|
"executor = LocalCheapExecutor()\n",
|
|
"inc_result = executor.evaluate(incumbent, fast_rung)\n",
|
|
"print(f\"Incumbent score: {inc_result.score:.4f}\\n\")\n",
|
|
"\n",
|
|
"# Evaluate neighbor challengers\n",
|
|
"scores = {}\n",
|
|
"for c in challengers[:8]:\n",
|
|
" result = executor.evaluate(c.spec, fast_rung)\n",
|
|
" scores[c.mutation_note] = result.score\n",
|
|
" marker = \">>>\" if result.score > inc_result.score else \" \"\n",
|
|
" print(f\" {marker} {c.mutation_note[:50]:50s} score={result.score:.4f}\")"
|
|
],
|
|
"outputs": [],
|
|
"execution_count": null
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Visualize\n",
|
|
"fig, ax = plt.subplots(figsize=(12, 5))\n",
|
|
"labels = [\"INCUMBENT\"] + [k[:35] for k in scores.keys()]\n",
|
|
"vals = [inc_result.score] + list(scores.values())\n",
|
|
"colors = [\"#e74c3c\"] + [\"#2ecc71\" if s > inc_result.score else \"#bdc3c7\" for s in scores.values()]\n",
|
|
"\n",
|
|
"ax.barh(range(len(labels)), vals, color=colors)\n",
|
|
"ax.set_yticks(range(len(labels)))\n",
|
|
"ax.set_yticklabels(labels, fontsize=8)\n",
|
|
"ax.axvline(x=inc_result.score, color=\"#e74c3c\", linestyle=\"--\", alpha=0.5)\n",
|
|
"ax.set_xlabel(\"Score\")\n",
|
|
"ax.set_title(\"Incumbent vs Challengers\")\n",
|
|
"ax.legend(handles=[Patch(color=\"#e74c3c\", label=\"Incumbent\"),\n",
|
|
" Patch(color=\"#2ecc71\", label=\"Beats incumbent\"),\n",
|
|
" Patch(color=\"#bdc3c7\", label=\"Below incumbent\")])\n",
|
|
"plt.tight_layout()\n",
|
|
"plt.show()"
|
|
],
|
|
"outputs": [],
|
|
"execution_count": null
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"---\n",
|
|
"## 6. One Ratchet Step in Detail\n",
|
|
"\n",
|
|
"Now let the harness orchestrate everything: generate challengers, evaluate, promote, select winner."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"metadata": {},
|
|
"source": [
|
|
"store = ResearchStore(tempfile.mkdtemp())\n",
|
|
"harness = AutoresearchHarness(store)\n",
|
|
"\n",
|
|
"step = harness.run_ratchet_step(fast_rung, allow_hardware=False)\n",
|
|
"\n",
|
|
"print(f\"Step index: {step.step_index}\")\n",
|
|
"print(f\"Incumbent before: {step.incumbent_before_id}\")\n",
|
|
"print(f\"Challengers tested: {len(step.challengers_tested)}\")\n",
|
|
"print(f\"Promoted: {len(step.promoted_challengers)}\")\n",
|
|
"print(f\"Winner: {step.winner_id}\")\n",
|
|
"print(f\"Winning margin: {step.winning_margin:+.4f}\")\n",
|
|
"print(f\"\\nCheap-tier: {step.cheap_tier_justification}\")\n",
|
|
"print(f\"\\nLesson: {step.distilled_lesson}\")"
|
|
],
|
|
"outputs": [],
|
|
"execution_count": null
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"metadata": {},
|
|
"source": [
|
|
"quiz(tracker, \"q5_no_winner\",\n",
|
|
" question=\"What happens if no challenger beats the incumbent?\",\n",
|
|
" options=[\n",
|
|
" \"The harness picks the best challenger anyway\",\n",
|
|
" \"The incumbent stays; the step is logged with zero improvement\",\n",
|
|
" \"The harness doubles the number of challengers\",\n",
|
|
" ],\n",
|
|
" correct=1, section=\"6. Ratchet step\", bloom=\"understand\",\n",
|
|
" explanation=\"Ratchet guarantee: the incumbent never gets worse. No-improvement steps are still valuable data for lesson extraction.\")"
|
|
],
|
|
"outputs": [],
|
|
"execution_count": null
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"---\n",
|
|
"## 7. Running a Full Rung with Patience\n",
|
|
"\n",
|
|
"A rung runs multiple steps. **Patience** stops early if no improvement is found."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"metadata": {},
|
|
"source": [
|
|
"store2 = ResearchStore(tempfile.mkdtemp())\n",
|
|
"harness2 = AutoresearchHarness(store2)\n",
|
|
"\n",
|
|
"multi_rung = RungConfig(\n",
|
|
" rung=1, name=\"Search Demo\", description=\"Full rung demo\",\n",
|
|
" objective=\"Find best config\",\n",
|
|
" bootstrap_incumbent=ExperimentSpec(\n",
|
|
" rung=1, target_backend=\"fake_brisbane\", noise_backend=\"fake_brisbane\",\n",
|
|
" shots=256, repeats=1,\n",
|
|
" ),\n",
|
|
" search_space=SearchSpaceConfig(\n",
|
|
" dimensions={\n",
|
|
" \"verification\": [\"both\", \"z_only\", \"x_only\"],\n",
|
|
" \"seed_style\": [\"h_p\", \"ry_rz\", \"u_magic\"],\n",
|
|
" \"postselection\": [\"all_measured\", \"z_only\", \"none\"],\n",
|
|
" },\n",
|
|
" max_challengers_per_step=6,\n",
|
|
" ),\n",
|
|
" tier_policy=TierPolicyConfig(\n",
|
|
" cheap_margin=0.001, cheap_shots=256, cheap_repeats=1,\n",
|
|
" promote_top_k=2, enable_hardware=False,\n",
|
|
" ),\n",
|
|
" score=rung_config.score,\n",
|
|
" step_budget=3, patience=2,\n",
|
|
" hardware=HardwareConfig(),\n",
|
|
")\n",
|
|
"\n",
|
|
"steps, lesson, feedback = harness2.run_rung(multi_rung, allow_hardware=False)\n",
|
|
"\n",
|
|
"print(f\"Steps completed: {len(steps)}\")\n",
|
|
"for s in steps:\n",
|
|
" improved = \"IMPROVED\" if s.winning_margin > 0 else \"no change\"\n",
|
|
" print(f\" Step {s.step_index}: margin={s.winning_margin:+.4f} ({improved})\")"
|
|
],
|
|
"outputs": [],
|
|
"execution_count": null
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"metadata": {},
|
|
"source": [
|
|
"quiz(tracker, \"q6_patience\",\n",
|
|
" question=\"Patience=2 means the rung stops after 2 consecutive steps with no improvement. Why?\",\n",
|
|
" options=[\n",
|
|
" \"To save memory\",\n",
|
|
" \"If 2 rounds of challengers all lose, the nearby parameter space is likely exhausted\",\n",
|
|
" \"2 is always the optimal patience value\",\n",
|
|
" ],\n",
|
|
" correct=1, section=\"7. Full rung\", bloom=\"evaluate\",\n",
|
|
" explanation=\"Patience prevents wasting compute once the search has converged. The budget is better spent on the next rung.\")\n",
|
|
"checkpoint_summary(tracker, \"7. Full rung\")"
|
|
],
|
|
"outputs": [],
|
|
"execution_count": null
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Score trajectory\n",
|
|
"experiments = store2.list_experiments(1)\n",
|
|
"exp_data = [(e[\"experiment_id\"][:20], e[\"final_score\"], e[\"role\"]) for e in experiments]\n",
|
|
"\n",
|
|
"fig, ax = plt.subplots(figsize=(12, 4))\n",
|
|
"x = range(len(exp_data))\n",
|
|
"colors = [\"#e74c3c\" if role == \"incumbent\" else \"#3498db\" for _, _, role in exp_data]\n",
|
|
"ax.bar(x, [s for _, s, _ in exp_data], color=colors)\n",
|
|
"ax.set_xlabel(\"Experiment\")\n",
|
|
"ax.set_ylabel(\"Score\")\n",
|
|
"ax.set_title(\"All Experiments in Rung\")\n",
|
|
"ax.legend(handles=[Patch(color=\"#e74c3c\", label=\"Incumbent\"), Patch(color=\"#3498db\", label=\"Challenger\")])\n",
|
|
"plt.tight_layout()\n",
|
|
"plt.show()"
|
|
],
|
|
"outputs": [],
|
|
"execution_count": null
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"---\n",
|
|
"## 8. Lesson Extraction\n",
|
|
"\n",
|
|
"After a rung, the harness analyzes all experiments and extracts **lessons** \u2014 both human-readable narratives and machine-readable rules."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"metadata": {},
|
|
"source": [
|
|
"print(\"=\" * 60)\n",
|
|
"print(lesson.narrative)\n",
|
|
"print(\"=\" * 60)"
|
|
],
|
|
"outputs": [],
|
|
"execution_count": null
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"metadata": {},
|
|
"source": [
|
|
"reflect(tracker, \"q7_lesson_quality\",\n",
|
|
" question=\"Read the lesson narrative. What actionable insight does it give? What would make it better?\",\n",
|
|
" section=\"8. Lessons\", bloom=\"evaluate\",\n",
|
|
" model_answer=\"A good lesson names specific values that helped/hurt and explains WHY. Machine-readable SearchRules are often more actionable than the narrative.\")"
|
|
],
|
|
"outputs": [],
|
|
"execution_count": null
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"metadata": {},
|
|
"source": [
|
|
"print(f\"\\nMachine-readable rules ({len(feedback.rules)}):\\n\")\n",
|
|
"for rule in feedback.rules:\n",
|
|
" print(f\" {rule.action.upper():7s} {rule.dimension}={rule.value}\")\n",
|
|
" print(f\" confidence={rule.confidence:.2f} reason: {rule.reason}\\n\")"
|
|
],
|
|
"outputs": [],
|
|
"execution_count": null
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"metadata": {},
|
|
"source": [
|
|
"quiz(tracker, \"q8_fix_vs_avoid\",\n",
|
|
" question=\"'fix' vs 'avoid' rules: what's the difference?\",\n",
|
|
" options=[\n",
|
|
" \"'fix' locks a value permanently; 'avoid' removes a value from the search space\",\n",
|
|
" \"'fix' repairs a bug; 'avoid' prevents a crash\",\n",
|
|
" \"They are synonyms\",\n",
|
|
" ],\n",
|
|
" correct=0, section=\"8. Lessons\", bloom=\"remember\",\n",
|
|
" explanation=\"'fix': always use this value. 'avoid': never use this value. Both narrow the search space for future rungs.\")\n",
|
|
"checkpoint_summary(tracker, \"8. Lessons\")"
|
|
],
|
|
"outputs": [],
|
|
"execution_count": null
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"---\n",
|
|
"## 9. LessonGuided Strategy\n",
|
|
"\n",
|
|
"Once we have rules, the **LessonGuided** strategy uses them to bias challenger generation toward promising regions."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"metadata": {},
|
|
"source": [
|
|
"if feedback.rules:\n",
|
|
" guided = LessonGuided(num_candidates=6)\n",
|
|
" guided_challengers = guided.generate(\n",
|
|
" incumbent, multi_rung.search_space, set(), [feedback]\n",
|
|
" )\n",
|
|
" print(f\"Lesson-guided: {len(guided_challengers)} challengers\")\n",
|
|
" for c in guided_challengers:\n",
|
|
" print(f\" {c.mutation_note}\")\n",
|
|
"else:\n",
|
|
" print(\"No rules extracted \u2014 try a larger step_budget for more data.\")"
|
|
],
|
|
"outputs": [],
|
|
"execution_count": null
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"---\n",
|
|
"## 10. Search Space Narrowing\n",
|
|
"\n",
|
|
"Rules can **narrow** the search space: remove \"avoid\" values, lock \"fix\" values."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"metadata": {},
|
|
"source": [
|
|
"print(\"BEFORE narrowing:\")\n",
|
|
"for dim, vals in multi_rung.search_space.dimensions.items():\n",
|
|
" print(f\" {dim}: {vals}\")\n",
|
|
"\n",
|
|
"narrowed = narrow_search_space(multi_rung.search_space, feedback.rules)\n",
|
|
"\n",
|
|
"print(\"\\nAFTER narrowing:\")\n",
|
|
"for dim, vals in narrowed.dimensions.items():\n",
|
|
" removed = set(multi_rung.search_space.dimensions[dim]) - set(vals)\n",
|
|
" suffix = f\" (removed: {removed})\" if removed else \"\"\n",
|
|
" print(f\" {dim}: {vals}{suffix}\")"
|
|
],
|
|
"outputs": [],
|
|
"execution_count": null
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"metadata": {},
|
|
"source": [
|
|
"quiz(tracker, \"q9_narrowing\",\n",
|
|
" question=\"What does search space narrowing accomplish?\",\n",
|
|
" options=[\n",
|
|
" \"It removes entire parameter dimensions\",\n",
|
|
" \"It removes poorly-performing values, keeping the dimension with fewer options\",\n",
|
|
" \"It adds new parameter values\",\n",
|
|
" ],\n",
|
|
" correct=1, section=\"10. Narrowing\", bloom=\"understand\",\n",
|
|
" explanation=\"Narrowing prunes bad values based on evidence. The dimension stays but with fewer options. A minimum is preserved to prevent overfitting.\")"
|
|
],
|
|
"outputs": [],
|
|
"execution_count": null
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"> **Key Insight:** Narrowing is \"learning\" \u2014 the machine prunes bad options based on evidence. This makes subsequent rungs faster and more focused.\n",
|
|
"\n",
|
|
"---\n",
|
|
"## 11. Cross-Rung Propagation\n",
|
|
"\n",
|
|
"A full **ratchet** chains multiple rungs. The winner from rung $N$ becomes the bootstrap for rung $N+1$."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"metadata": {},
|
|
"source": [
|
|
"store3 = ResearchStore(tempfile.mkdtemp())\n",
|
|
"harness3 = AutoresearchHarness(store3)\n",
|
|
"\n",
|
|
"rung1 = RungConfig(\n",
|
|
" rung=1, name=\"Rung 1\", description=\"Explore basics\",\n",
|
|
" objective=\"Find best seed and verification\",\n",
|
|
" bootstrap_incumbent=ExperimentSpec(\n",
|
|
" rung=1, target_backend=\"fake_brisbane\", noise_backend=\"fake_brisbane\",\n",
|
|
" shots=256, repeats=1,\n",
|
|
" ),\n",
|
|
" search_space=SearchSpaceConfig(\n",
|
|
" dimensions={\"verification\": [\"both\", \"z_only\"], \"seed_style\": [\"h_p\", \"ry_rz\"]},\n",
|
|
" max_challengers_per_step=4,\n",
|
|
" ),\n",
|
|
" tier_policy=TierPolicyConfig(cheap_margin=0.0, cheap_shots=256, cheap_repeats=1,\n",
|
|
" promote_top_k=1, enable_hardware=False),\n",
|
|
" score=rung_config.score, step_budget=2, patience=1, hardware=HardwareConfig(),\n",
|
|
")\n",
|
|
"\n",
|
|
"rung2 = RungConfig(\n",
|
|
" rung=2, name=\"Rung 2\", description=\"Refine optimization\",\n",
|
|
" objective=\"Tune optimization level\",\n",
|
|
" bootstrap_incumbent=ExperimentSpec(\n",
|
|
" rung=2, target_backend=\"fake_brisbane\", noise_backend=\"fake_brisbane\",\n",
|
|
" shots=256, repeats=1,\n",
|
|
" ),\n",
|
|
" search_space=SearchSpaceConfig(\n",
|
|
" dimensions={\"optimization_level\": [1, 2, 3], \"verification\": [\"both\", \"z_only\"]},\n",
|
|
" max_challengers_per_step=4,\n",
|
|
" ),\n",
|
|
" tier_policy=rung1.tier_policy,\n",
|
|
" score=rung_config.score, step_budget=2, patience=1, hardware=HardwareConfig(),\n",
|
|
")\n",
|
|
"\n",
|
|
"results = harness3.run_ratchet([rung1, rung2], allow_hardware=False)\n",
|
|
"\n",
|
|
"for lesson_obj, fb in results:\n",
|
|
" print(f\"\\nRung {lesson_obj.rung}: {lesson_obj.name}\")\n",
|
|
" print(f\" Rules: {len(fb.rules)}\")\n",
|
|
" best_fields = {k: v for k, v in list(fb.best_spec_fields.items())[:4]}\n",
|
|
" print(f\" Best spec: {best_fields}...\")\n",
|
|
"\n",
|
|
"print(f\"\\nAccumulated lessons: {len(harness3._accumulated_lessons)} rungs of feedback\")"
|
|
],
|
|
"outputs": [],
|
|
"execution_count": null
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"---\n",
|
|
"## 12. Transfer Evaluation\n",
|
|
"\n",
|
|
"A transfer test checks if the best settings generalize across different backend noise profiles (not overfit to one)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"metadata": {},
|
|
"source": [
|
|
"evaluator = TransferEvaluator()\n",
|
|
"report = evaluator.evaluate_across_backends(\n",
|
|
" incumbent,\n",
|
|
" [\"fake_brisbane\"], # Use single backend for speed\n",
|
|
" fast_rung,\n",
|
|
")\n",
|
|
"print(f\"Transfer score (pessimistic = min): {report.transfer_score:.4f}\")\n",
|
|
"print(f\"Mean score: {report.mean_score:.4f}\")\n",
|
|
"for name, score in report.per_backend_scores.items():\n",
|
|
" print(f\" {name}: {score:.4f}\")"
|
|
],
|
|
"outputs": [],
|
|
"execution_count": null
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"metadata": {},
|
|
"source": [
|
|
"quiz(tracker, \"q10_transfer\",\n",
|
|
" question=\"A spec scores 0.8 on one backend but 0.3 on another. What does this mean?\",\n",
|
|
" options=[\n",
|
|
" \"The spec is bad overall\",\n",
|
|
" \"The spec is overfitted to the first backend's noise profile\",\n",
|
|
" \"The second backend is broken\",\n",
|
|
" ],\n",
|
|
" correct=1, section=\"12. Transfer\", bloom=\"evaluate\",\n",
|
|
" explanation=\"A large transfer drop means settings are tuned to one backend's quirks. The ratchet tests transfer to find robust, generalizable configurations.\")\n",
|
|
"checkpoint_summary(tracker, \"12. Transfer\")"
|
|
],
|
|
"outputs": [],
|
|
"execution_count": null
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"---\n",
|
|
"## Summary\n",
|
|
"\n",
|
|
"| Concept | What it does |\n",
|
|
"|---|---|\n",
|
|
"| **NeighborWalk** | Single-axis mutations \u2014 systematic but limited |\n",
|
|
"| **RandomCombo** | Multi-axis mutations \u2014 discovers interactions |\n",
|
|
"| **LessonGuided** | Rule-biased mutations \u2014 focuses on promising regions |\n",
|
|
"| **Ratchet step** | Generate, evaluate, promote, select winner |\n",
|
|
"| **Patience** | Stop early if no improvement |\n",
|
|
"| **Lesson extraction** | Human-readable + machine-readable rules from data |\n",
|
|
"| **Search narrowing** | Prune bad values, lock good ones |\n",
|
|
"| **Cross-rung propagation** | Winner and lessons flow to the next rung |\n",
|
|
"| **Transfer evaluation** | Check generalization across noise profiles |\n",
|
|
"\n",
|
|
"> **Dashboard Exercise:** Try to manually find the best configuration in `00_dashboard.ipynb`. Then compare your best score to what the ratchet found. Who wins?"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"---\n",
|
|
"## Final Assessment"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"metadata": {},
|
|
"source": [
|
|
"tracker.dashboard()\n",
|
|
"path = tracker.save()\n",
|
|
"print(f\"\\nProgress saved to: {path}\")"
|
|
],
|
|
"outputs": [],
|
|
"execution_count": null
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "1cc666e4",
|
|
"source": "---\n## You've completed Plan C!\n\nWant to explore the same material from a different angle? Try another plan:\n- [Plan A \u2014 Sequential](../plan_a/01_encoded_magic_state.ipynb) (step-by-step, three notebooks)\n- [Plan B \u2014 Spiral Notebook](../plan_b/spiral_notebook.ipynb) (three passes, increasing depth)\n- [Plan D \u2014 Hypothesis-Driven](../plan_d/experiment_1_protection.ipynb) (experimental method)\n\n*\u2190 [Dashboard](00_dashboard.ipynb) \u00b7 [Track B \u2014 Engineering](track_b_engineering.ipynb) \u00b7 [Start Here](../00_START_HERE.ipynb)*",
|
|
"metadata": {}
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"name": "python",
|
|
"version": "3.14.2"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
} |