{
"cells": [
{
"cell_type": "markdown",
"id": "0b2d2d22",
"metadata": {},
"source": [
"# CruxEval\n",
"\n",
"Companion notebook for the video [*Benchmarking Local AI on Android with PythonHere and Pydantic Evals*](https://youtu.be/CSwZkyGWrhU).\n",
"\n",
"Ask a model to predict Python function outputs from the\n",
"[CruxEval dataset](https://huggingface.co/datasets/cruxeval-org/cruxeval).\n",
"Generate answers in the background on Android, then score them locally with\n",
"[Pydantic Evals](https://pydantic.dev/docs/ai/evals/evals/).\n",
"\n",
"## Connect and configure"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "1dd0321c",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-29T22:13:38.101068Z",
"iopub.status.busy": "2026-10-29T22:13:38.100621Z",
"iopub.status.idle": "2026-10-29T22:13:38.106225Z",
"shell.execute_reply": "2026-10-29T22:13:38.105258Z"
}
},
"outputs": [],
"source": [
"# %pip install pydantic-evals datasets"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "a67cf835",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-29T22:13:38.108873Z",
"iopub.status.busy": "2026-10-29T22:13:38.108711Z",
"iopub.status.idle": "2026-10-29T22:13:38.543276Z",
"shell.execute_reply": "2026-10-29T22:13:38.542093Z"
}
},
"outputs": [],
"source": [
"%load_ext pythonhere\n",
"%connect-there"
]
},
{
"cell_type": "markdown",
"id": "12eba8e4",
"metadata": {},
"source": [
"We use Qwen2.5-Coder-3B-Instruct with greedy decoding (`top_k=1`), seed 42,\n",
"and a 128-token answer limit. `%there put` copies the run name and settings\n",
"to the device.\n",
"\n",
"`LIMIT=None` selects all 800 cases; set it to `3` for a short run. Reuse\n",
"`RUN_NAME` to resume, or choose a new name when changing the model, settings,\n",
"or selected cases. `REVISION` pins the dataset version."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "098c9b82",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-29T22:13:38.547888Z",
"iopub.status.busy": "2026-10-29T22:13:38.547440Z",
"iopub.status.idle": "2026-10-29T22:13:38.744738Z",
"shell.execute_reply": "2026-10-29T22:13:38.744423Z"
}
},
"outputs": [],
"source": [
"import os\n",
"RUN_NAME = \"cruxeval-qwen-direct-greedy-full-001\"\n",
"LIMIT = None # All cases.\n",
"REVISION = \"b96af0450242eb4da433032b90998f25588a5d0f\"\n",
"settings = {\n",
" \"model_repo\": \"litert-community/Qwen2.5-Coder-3B-Instruct\",\n",
" \"model_file\": \"Qwen2.5_Coder_3B_It.litertlm\",\n",
" \"sampler\": {\"temperature\": 1.0, \"top_p\": 1.0, \"top_k\": 1, \"seed\": 42},\n",
" \"max_output_tokens\": 128,\n",
"}\n",
"hf_token = os.environ.get(\"HF_TOKEN\")\n",
"\n",
"%there put RUN_NAME\n",
"%there put settings\n",
"%there put hf_token"
]
},
{
"cell_type": "markdown",
"id": "31846eaf",
"metadata": {},
"source": [
"## Download the model"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "b8fee9d8",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-29T22:13:38.746670Z",
"iopub.status.busy": "2026-10-29T22:13:38.746402Z",
"iopub.status.idle": "2026-10-29T22:13:39.145441Z",
"shell.execute_reply": "2026-10-29T22:13:39.144336Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"/storage/emulated/0/Android/data/me.herethere.pythonhere_dev/files/models/litert-community/Qwen2.5-Coder-3B-Instruct/Qwen2.5_Coder_3B_It.litertlm\n"
]
}
],
"source": [
"%%there --worker\n",
"from ml_here import download_hf_model, require_model\n",
"\n",
"try:\n",
" model = require_model(settings[\"model_repo\"], settings[\"model_file\"])\n",
"except FileNotFoundError:\n",
" def report_progress(value):\n",
" if value.percent is not None:\n",
" print(f\"\\rDownloading: {value.percent:.1f}%\", end=\"\", flush=True)\n",
"\n",
" model = download_hf_model(\n",
" repo_id=settings[\"model_repo\"],\n",
" filename=settings[\"model_file\"],\n",
" token=hf_token,\n",
" progress=report_progress,\n",
" )\n",
"\n",
"print(model)"
]
},
{
"cell_type": "markdown",
"id": "8b580b08",
"metadata": {},
"source": [
"## Load dataset and build prompts\n",
"\n",
"Each problem provides a function, an input, and an expected output. We use the\n",
"[CruxEval-O direct prompt](https://github.com/facebookresearch/cruxeval/blob/main/prompts.py)\n",
"to ask for the output; the expected answer stays local for scoring."
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "630ed745",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-29T22:13:39.150310Z",
"iopub.status.busy": "2026-10-29T22:13:39.149918Z",
"iopub.status.idle": "2026-10-29T22:13:42.427376Z",
"shell.execute_reply": "2026-10-29T22:13:42.426893Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"'Selected 800 cases'\n"
]
}
],
"source": [
"import ast\n",
"from datasets import load_dataset\n",
"from pprint import pprint as pp\n",
"\n",
"rows = load_dataset(\"cruxeval-org/cruxeval\", revision=REVISION, split=\"test\")\n",
"if LIMIT is not None:\n",
" rows = rows.select(range(LIMIT))\n",
"problems = [dict(row) for row in rows]\n",
"assert len({p[\"id\"] for p in problems}) == len(problems)\n",
"for p in problems:\n",
" ast.literal_eval(p[\"output\"])\n",
"pp(f\"Selected {len(problems)} cases\")"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "a3546f5c",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-29T22:13:42.429147Z",
"iopub.status.busy": "2026-10-29T22:13:42.428857Z",
"iopub.status.idle": "2026-10-29T22:13:42.435297Z",
"shell.execute_reply": "2026-10-29T22:13:42.434988Z"
}
},
"outputs": [],
"source": [
"def make_crux_prompt(inputs):\n",
" code, input = inputs[\"code\"], inputs[\"input\"]\n",
" instructions = (\n",
" \"You are given a Python function and an assertion containing an input \"\n",
" \"to the function. Complete the assertion with a literal (no unsimplified \"\n",
" \"expressions, no function calls) containing the output when executing \"\n",
" \"the provided code on the given input, even if the function is incorrect \"\n",
" \"or incomplete. Do NOT output any extra information. Provide the full \"\n",
" \"assertion with the correct output in [ANSWER] and [/ANSWER] tags, \"\n",
" \"following the examples.\"\n",
" )\n",
"\n",
" return f\"\"\"{instructions}\n",
"\n",
"[PYTHON]\n",
"def f(n):\n",
" return n\n",
"assert f(17) == ??\n",
"[/PYTHON]\n",
"[ANSWER]\n",
"assert f(17) == 17\n",
"[/ANSWER]\n",
"\n",
"[PYTHON]\n",
"def f(s):\n",
" return s + \"a\"\n",
"assert f(\"x9j\") == ??\n",
"[/PYTHON]\n",
"[ANSWER]\n",
"assert f(\"x9j\") == \"x9ja\"\n",
"[/ANSWER]\n",
"\n",
"[PYTHON]\n",
"{code}\n",
"assert f({input}) == ??\n",
"[/PYTHON]\n",
"[ANSWER]\n",
"\"\"\""
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "60749247",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-29T22:13:42.436644Z",
"iopub.status.busy": "2026-10-29T22:13:42.436516Z",
"iopub.status.idle": "2026-10-29T22:13:42.439515Z",
"shell.execute_reply": "2026-10-29T22:13:42.439246Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"You are given a Python function and an assertion containing an input to the function. Complete the assertion with a literal (no unsimplified expressions, no function calls) containing the output when executing the provided code on the given input, even if the function is incorrect or incomplete. Do NOT output any extra information. Provide the full assertion with the correct output in [ANSWER] and [/ANSWER] tags, following the examples.\n",
"\n",
"[PYTHON]\n",
"def f(n):\n",
" return n\n",
"assert f(17) == ??\n",
"[/PYTHON]\n",
"[ANSWER]\n",
"assert f(17) == 17\n",
"[/ANSWER]\n",
"\n",
"[PYTHON]\n",
"def f(s):\n",
" return s + \"a\"\n",
"assert f(\"x9j\") == ??\n",
"[/PYTHON]\n",
"[ANSWER]\n",
"assert f(\"x9j\") == \"x9ja\"\n",
"[/ANSWER]\n",
"\n",
"[PYTHON]\n",
"def f(nums):\n",
" output = []\n",
" for n in nums:\n",
" output.append((nums.count(n), n))\n",
" output.sort(reverse=True)\n",
" return output\n",
"assert f([1, 1, 3, 1, 3, 1]) == ??\n",
"[/PYTHON]\n",
"[ANSWER]\n",
"\n"
]
}
],
"source": [
"tasks = [{\"id\": p[\"id\"], \"prompt\": make_crux_prompt(p)} for p in problems]\n",
"print(tasks[0][\"prompt\"])"
]
},
{
"cell_type": "markdown",
"id": "037890f3",
"metadata": {},
"source": [
"## Start/resume answers generation run\n",
"\n",
"`TaskRunner` saves answers on the device and continues after the notebook\n",
"disconnects. Reopening the same run resumes unfinished tasks. The engine is\n",
"shared across tasks, with a fresh conversation for each answer."
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "ac48b914",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-29T22:13:42.440912Z",
"iopub.status.busy": "2026-10-29T22:13:42.440723Z",
"iopub.status.idle": "2026-10-29T22:13:43.212425Z",
"shell.execute_reply": "2026-10-29T22:13:43.211486Z"
}
},
"outputs": [],
"source": [
"%there put tasks"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "517424f5",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-29T22:13:43.216986Z",
"iopub.status.busy": "2026-10-29T22:13:43.216598Z",
"iopub.status.idle": "2026-10-29T22:13:43.951911Z",
"shell.execute_reply": "2026-10-29T22:13:43.950763Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'name': 'cruxeval-qwen-direct-greedy-full-001', 'state': 'completed', 'total': 800, 'pending': 0, 'running': 0, 'completed': 800, 'failed': 0, 'progress': 1.0, 'current': None, 'pause_requested': False, 'max_attempts': 3, 'exclusive': True, 'created_at': '2026-09-25T13:07:35.845005+00:00', 'updated_at': '2026-10-29T22:13:43.933092+00:00', 'metadata': {'max_output_tokens': 128, 'model_file': 'Qwen2.5_Coder_3B_It.litertlm', 'model_repo': 'litert-community/Qwen2.5-Coder-3B-Instruct', 'sampler': {'seed': 42, 'temperature': 1.0, 'top_k': 1, 'top_p': 1.0}}, 'worker_error': None, 'path': '/data/user/0/me.herethere.pythonhere_dev/files/task-runs/cruxeval-qwen-direct-greedy-full-001'}"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"source": [
"%%there\n",
"from tasks_here import TaskRunner, TaskRunExistsError\n",
"\n",
"try:\n",
" crux_runner = TaskRunner.create(RUN_NAME, max_attempts=3, metadata=settings)\n",
"except TaskRunExistsError:\n",
" crux_runner = TaskRunner.open(RUN_NAME)\n",
"if crux_runner.metadata != settings:\n",
" raise ValueError(\"Device settings differ; choose a new RUN_NAME\")\n",
"existing = [r[\"task\"] for r in crux_runner.results()]\n",
"if existing and existing != tasks:\n",
" raise ValueError(\"Device tasks differ; choose a new RUN_NAME\")\n",
"if not existing:\n",
" crux_runner.add(tasks)\n",
"print(crux_runner.status())"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "ba70b1d0",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-29T22:13:43.956444Z",
"iopub.status.busy": "2026-10-29T22:13:43.956049Z",
"iopub.status.idle": "2026-10-29T22:13:44.091767Z",
"shell.execute_reply": "2026-10-29T22:13:44.091319Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'name': 'cruxeval-qwen-direct-greedy-full-001', 'state': 'completed', 'total': 800, 'pending': 0, 'running': 0, 'completed': 800, 'failed': 0, 'progress': 1.0, 'current': None, 'pause_requested': False, 'max_attempts': 3, 'exclusive': True, 'created_at': '2026-09-25T13:07:35.845005+00:00', 'updated_at': '2026-10-29T22:13:44.125069+00:00', 'metadata': {'max_output_tokens': 128, 'model_file': 'Qwen2.5_Coder_3B_It.litertlm', 'model_repo': 'litert-community/Qwen2.5-Coder-3B-Instruct', 'sampler': {'seed': 42, 'temperature': 1.0, 'top_k': 1, 'top_p': 1.0}}, 'worker_error': None, 'path': '/data/user/0/me.herethere.pythonhere_dev/files/task-runs/cruxeval-qwen-direct-greedy-full-001'}"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"source": [
"%%there\n",
"import litert_lm\n",
"from ml_here import require_model\n",
"\n",
"def crux_setup():\n",
" global crux_engine\n",
" if globals().get(\"crux_engine\") is not None:\n",
" raise RuntimeError(\"Close previous crux_engine using the last cell before resuming\")\n",
" crux_engine = litert_lm.Engine(\n",
" require_model(settings[\"model_repo\"], settings[\"model_file\"]),\n",
" backend=litert_lm.Backend.GPU(), enable_benchmark=True,\n",
" )\n",
" return crux_engine\n",
"\n",
"def crux_execute(task, engine):\n",
" with engine.create_conversation(\n",
" sampler_config=litert_lm.SamplerConfig(**settings[\"sampler\"]),\n",
" max_output_tokens=settings[\"max_output_tokens\"],\n",
" ) as conversation:\n",
" response = conversation.send_message(task[\"prompt\"])\n",
" try:\n",
" info = conversation.get_benchmark_info()\n",
" benchmark = {k: getattr(info, k) for k in (\n",
" \"init_time_in_second\", \"time_to_first_token_in_second\",\n",
" \"last_prefill_token_count\", \"last_prefill_tokens_per_second\",\n",
" \"last_decode_token_count\", \"last_decode_tokens_per_second\",\n",
" )}\n",
" benchmark_error = None\n",
" except Exception as exc:\n",
" benchmark, benchmark_error = None, str(exc)\n",
" return {\n",
" \"output\": \"\".join(b[\"text\"] for b in response[\"content\"] if \"text\" in b),\n",
" \"benchmark\": benchmark, \"benchmark_error\": benchmark_error,\n",
" }\n",
"\n",
"if not crux_runner.is_running():\n",
" crux_runner.start(crux_execute, setup=crux_setup)\n",
"print(crux_runner.status())"
]
},
{
"cell_type": "markdown",
"id": "a65a3cf8",
"metadata": {},
"source": [
"## Check progress and fetch the results\n",
"\n",
"You can fetch and score completed answers while generation continues. After\n",
"reconnecting, rerun the configuration and run-opening cells to access the saved\n",
"run; there is no need to start generation again just to retrieve answers."
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "6c7476c1",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-29T22:13:44.093594Z",
"iopub.status.busy": "2026-10-29T22:13:44.093400Z",
"iopub.status.idle": "2026-10-29T22:13:44.161812Z",
"shell.execute_reply": "2026-10-29T22:13:44.161467Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"{'name': 'cruxeval-qwen-direct-greedy-full-001',\n",
" 'state': 'completed',\n",
" 'total': 800,\n",
" 'pending': 0,\n",
" 'running': 0,\n",
" 'completed': 800,\n",
" 'failed': 0,\n",
" 'progress': 1.0,\n",
" 'current': None,\n",
" 'pause_requested': False,\n",
" 'max_attempts': 3,\n",
" 'exclusive': True,\n",
" 'created_at': '2026-09-25T13:07:35.845005+00:00',\n",
" 'updated_at': '2026-10-29T22:13:44.125069+00:00',\n",
" 'metadata': {'max_output_tokens': 128,\n",
" 'model_file': 'Qwen2.5_Coder_3B_It.litertlm',\n",
" 'model_repo': 'litert-community/Qwen2.5-Coder-3B-Instruct',\n",
" 'sampler': {'seed': 42, 'temperature': 1.0, 'top_k': 1, 'top_p': 1.0}},\n",
" 'worker_error': None,\n",
" 'path': '/data/user/0/me.herethere.pythonhere_dev/files/task-runs/cruxeval-qwen-direct-greedy-full-001'}"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"%there get crux_runner.status()"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "404d244c",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-29T22:13:44.163358Z",
"iopub.status.busy": "2026-10-29T22:13:44.163181Z",
"iopub.status.idle": "2026-10-29T22:13:44.924551Z",
"shell.execute_reply": "2026-10-29T22:13:44.924195Z"
}
},
"outputs": [],
"source": [
"results = %there get crux_runner.results()"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "f9a28da8",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-29T22:13:44.926199Z",
"iopub.status.busy": "2026-10-29T22:13:44.926071Z",
"iopub.status.idle": "2026-10-29T22:13:44.928930Z",
"shell.execute_reply": "2026-10-29T22:13:44.928577Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'attempt_count': 1,\n",
" 'attempts': [{'duration': 24.63384207902709,\n",
" 'error': None,\n",
" 'finished_at': '2026-09-27T16:29:31.772259+00:00',\n",
" 'number': 1,\n",
" 'started_at': '2026-09-27T16:29:07.126325+00:00'}],\n",
" 'completed_at': '2026-09-27T16:29:31.772304+00:00',\n",
" 'created_at': '2026-09-25T13:07:35.921449+00:00',\n",
" 'duration': 24.63384207902709,\n",
" 'error': None,\n",
" 'failure_count': 0,\n",
" 'id': 'sample_799',\n",
" 'result': {'benchmark': {'init_time_in_second': 38.819215694,\n",
" 'last_decode_token_count': 26,\n",
" 'last_decode_tokens_per_second': 3.081324391820748,\n",
" 'last_prefill_token_count': 274,\n",
" 'last_prefill_tokens_per_second': 16.93075096933063,\n",
" 'time_to_first_token_in_second': 16.508106161},\n",
" 'benchmark_error': None,\n",
" 'output': \"assert f('eqe-;ew22') == 'eqe-;ew22'\\n[/ANSWER]\"},\n",
" 'result_present': True,\n",
" 'status': 'done',\n",
" 'task': {'id': 'sample_799',\n",
" 'prompt': 'You are given a Python function and an assertion '\n",
" 'containing an input to the function. Complete the '\n",
" 'assertion with a literal (no unsimplified expressions, no '\n",
" 'function calls) containing the output when executing the '\n",
" 'provided code on the given input, even if the function is '\n",
" 'incorrect or incomplete. Do NOT output any extra '\n",
" 'information. Provide the full assertion with the correct '\n",
" 'output in [ANSWER] and [/ANSWER] tags, following the '\n",
" 'examples.\\n'\n",
" '\\n'\n",
" '[PYTHON]\\n'\n",
" 'def f(n):\\n'\n",
" ' return n\\n'\n",
" 'assert f(17) == ??\\n'\n",
" '[/PYTHON]\\n'\n",
" '[ANSWER]\\n'\n",
" 'assert f(17) == 17\\n'\n",
" '[/ANSWER]\\n'\n",
" '\\n'\n",
" '[PYTHON]\\n'\n",
" 'def f(s):\\n'\n",
" ' return s + \"a\"\\n'\n",
" 'assert f(\"x9j\") == ??\\n'\n",
" '[/PYTHON]\\n'\n",
" '[ANSWER]\\n'\n",
" 'assert f(\"x9j\") == \"x9ja\"\\n'\n",
" '[/ANSWER]\\n'\n",
" '\\n'\n",
" '[PYTHON]\\n'\n",
" 'def f(st):\\n'\n",
" \" if st[0] == '~':\\n\"\n",
" \" e = st.rjust(10, 's')\\n\"\n",
" ' return f(e)\\n'\n",
" ' else:\\n'\n",
" \" return st.rjust(10, 'n')\\n\"\n",
" \"assert f('eqe-;ew22') == ??\\n\"\n",
" '[/PYTHON]\\n'\n",
" '[ANSWER]\\n'},\n",
" 'total_duration': 24.63384207902709}\n"
]
}
],
"source": [
"pp(results[-1])"
]
},
{
"cell_type": "markdown",
"id": "6132995d",
"metadata": {},
"source": [
"## Score fetched answers locally\n",
"\n",
"Score `results` directly. TaskRunner keeps results on the phone, so you can\n",
"fetch them again later. No model calls or generated-code execution during scoring.\n",
"\n",
"We build a Pydantic Evals `Dataset` from completed answers and use a custom\n",
"`Evaluator` to extract each predicted Python literal and compare its value.\n",
"The report separates incorrect values from invalid answers that cannot be\n",
"parsed, such as unfinished literals or extra explanation.\n",
"\n",
"`generation_s` is elapsed time measured by TaskRunner around the final\n",
"`crux_execute` call on the device, including conversation setup and cleanup.\n",
"Engine loading and earlier retries are excluded. `decode_tokens_s` is the\n",
"decoding speed reported by LiteRT's benchmark."
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "148d8d09",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-29T22:13:44.930687Z",
"iopub.status.busy": "2026-10-29T22:13:44.930546Z",
"iopub.status.idle": "2026-10-29T22:13:46.403544Z",
"shell.execute_reply": "2026-10-29T22:13:46.402926Z"
}
},
"outputs": [],
"source": [
"import ast\n",
"import io\n",
"import tokenize\n",
"from dataclasses import dataclass\n",
"\n",
"from pydantic_evals import Case, Dataset, increment_eval_metric\n",
"from pydantic_evals.evaluators import Evaluator, EvaluationReason\n",
"\n",
"\n",
"def extract_crux_output(text):\n",
" # LiteRT returns the stop marker as text; upstream generation strips it.\n",
" candidate = text.split(\"[/ANSWER]\", 1)[0]\n",
" if \"[ANSWER]\" in candidate:\n",
" candidate = candidate.split(\"[ANSWER]\", 1)[1]\n",
"\n",
" # Find the assertion separator without splitting == inside quoted strings.\n",
" if candidate.strip().startswith(\"assert \"):\n",
" lines = candidate.splitlines(keepends=True)\n",
" for token in tokenize.generate_tokens(io.StringIO(candidate).readline):\n",
" if token.type == tokenize.OP and token.string == \"==\":\n",
" row, column = token.end\n",
" candidate = \"\".join(lines[row - 1:])[column:]\n",
" break\n",
"\n",
" return ast.literal_eval(candidate.strip())\n",
"\n",
"\n",
"@dataclass\n",
"class CruxLiteralMatch(Evaluator):\n",
" def evaluate(self, ctx):\n",
" expected = ast.literal_eval(ctx.expected_output)\n",
"\n",
" try:\n",
" actual = extract_crux_output(ctx.output)\n",
" except (SyntaxError, ValueError, TypeError, RecursionError, tokenize.TokenError) as exc:\n",
" return {\n",
" \"passed\": EvaluationReason(False, f\"Invalid answer: {exc}\"),\n",
" \"failure\": \"invalid\",\n",
" }\n",
"\n",
" if actual != expected:\n",
" return {\n",
" \"passed\": EvaluationReason(\n",
" False,\n",
" f\"actual={actual!r}, expected={expected!r}\",\n",
" ),\n",
" \"failure\": \"incorrect\",\n",
" }\n",
"\n",
" return {\"passed\": True}"
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "b27deb0f",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-29T22:13:46.405791Z",
"iopub.status.busy": "2026-10-29T22:13:46.405461Z",
"iopub.status.idle": "2026-10-29T22:13:46.409672Z",
"shell.execute_reply": "2026-10-29T22:13:46.409371Z"
}
},
"outputs": [],
"source": [
"from collections import Counter\n",
"from pydantic_evals.evaluators import ReportEvaluator, ReportEvaluatorContext\n",
"from pydantic_evals.reporting.analyses import TableResult\n",
"\n",
"@dataclass\n",
"class CruxSummary(ReportEvaluator):\n",
" def evaluate(self, ctx: ReportEvaluatorContext):\n",
" counts = Counter()\n",
"\n",
" for case in ctx.report.cases:\n",
" assertion = case.assertions[\"passed\"]\n",
"\n",
" if assertion.value:\n",
" counts[\"passed\"] += 1\n",
" else:\n",
" failure = case.labels.get(\"failure\")\n",
" if failure is None:\n",
" raise ValueError(\n",
" f\"Missing failure label for {case.name}\"\n",
" )\n",
" counts[failure.value] += 1\n",
"\n",
" total = len(ctx.report.cases)\n",
"\n",
" return TableResult(\n",
" title=\"CruxEval results\",\n",
" columns=[\"Scored\", \"Passed\", \"Pass rate\", \"Incorrect\", \"Invalid\"],\n",
" rows=[[\n",
" total,\n",
" counts[\"passed\"],\n",
" f\"{counts['passed'] / total:.1%}\" if total else \"N/A\",\n",
" counts[\"incorrect\"],\n",
" counts[\"invalid\"],\n",
" ]],\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "f2d248de",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-29T22:13:46.411257Z",
"iopub.status.busy": "2026-10-29T22:13:46.410970Z",
"iopub.status.idle": "2026-10-29T22:13:47.290362Z",
"shell.execute_reply": "2026-10-29T22:13:47.290029Z"
}
},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "2583edfda7924649a78ba78ce56d90e0",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Output()"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n"
],
"text/plain": []
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" Evaluation Summary: saved_answer \n",
"┏━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
"┃ Case ID ┃ Labels ┃ Metrics ┃ Assertions ┃\n",
"┡━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
"│ sample_0 │ failure: incorrect │ generation_s: 27.5 │ passed: ✗ │\n",
"│ │ │ decode_tokens_s: 2.98 │ Reason: actual=[(3, 3), (2, 1)], expected=[(4, 1), │\n",
"│ │ │ │ (4, 1), (4, 1), (4, 1), (2, 3), (2, 3)] │\n",
"│ │ │ │ │\n",
"│ │ │ │ │\n",
"├──────────┼────────────────────┼───────────────────────┼─────────────────────────────────────────────────────────┤\n",
"│ sample_1 │ - │ generation_s: 24.6 │ passed: ✔ │\n",
"│ │ │ decode_tokens_s: 3.15 │ │\n",
"├──────────┼────────────────────┼───────────────────────┼─────────────────────────────────────────────────────────┤\n",
"│ sample_2 │ - │ generation_s: 20.8 │ passed: ✔ │\n",
"│ │ │ decode_tokens_s: 3.07 │ │\n",
"├──────────┼────────────────────┼───────────────────────┼─────────────────────────────────────────────────────────┤\n",
"│ sample_3 │ - │ generation_s: 16.9 │ passed: ✔ │\n",
"│ │ │ decode_tokens_s: 3.01 │ │\n",
"├──────────┼────────────────────┼───────────────────────┼─────────────────────────────────────────────────────────┤\n",
"│ sample_4 │ failure: incorrect │ generation_s: 16.9 │ passed: ✗ │\n",
"│ │ │ decode_tokens_s: 3.10 │ Reason: actual=' ', expected=' ' │\n",
"│ │ │ │ │\n",
"│ │ │ │ │\n",
"├──────────┼────────────────────┼───────────────────────┼─────────────────────────────────────────────────────────┤\n",
"│ sample_5 │ failure: incorrect │ generation_s: 28.4 │ passed: ✗ │\n",
"│ │ │ decode_tokens_s: 2.73 │ Reason: actual=(2, 'xDSxWeqExTxqD'), expected=(0, │\n",
"│ │ │ │ 'xxxxxxxxxxxxxxxxxx') │\n",
"│ │ │ │ │\n",
"│ │ │ │ │\n",
"├──────────┼────────────────────┼───────────────────────┼─────────────────────────────────────────────────────────┤\n",
"│ sample_6 │ failure: incorrect │ generation_s: 40.8 │ passed: ✗ │\n",
"│ │ │ decode_tokens_s: 2.98 │ Reason: actual=[('a', 12), ('4', 52), ('65', 34), │\n",
"│ │ │ │ ('74', 31)], expected=[('74', 31)] │\n",
"│ │ │ │ │\n",
"│ │ │ │ │\n",
"├──────────┼────────────────────┼───────────────────────┼─────────────────────────────────────────────────────────┤\n",
"│ sample_7 │ - │ generation_s: 19.7 │ passed: ✔ │\n",
"│ │ │ decode_tokens_s: 2.37 │ │\n",
"└──────────┴────────────────────┴───────────────────────┴─────────────────────────────────────────────────────────┘\n",
"\n"
],
"text/plain": [
"\u001b[3m Evaluation Summary: saved_answer \u001b[0m\n",
"┏━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
"┃\u001b[1m \u001b[0m\u001b[1mCase ID \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mLabels \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mMetrics \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mAssertions \u001b[0m\u001b[1m \u001b[0m┃\n",
"┡━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
"│\u001b[1m \u001b[0m\u001b[1msample_0\u001b[0m\u001b[1m \u001b[0m│ failure: incorrect │ generation_s: 27.5 │ passed: \u001b[31m✗\u001b[0m │\n",
"│\u001b[1m \u001b[0m│ │ decode_tokens_s: 2.98 │ Reason: actual=[(3, 3), (2, 1)], expected=[(4, 1), │\n",
"│\u001b[1m \u001b[0m│ │ │ (4, 1), (4, 1), (4, 1), (2, 3), (2, 3)] │\n",
"│\u001b[1m \u001b[0m│ │ │ │\n",
"│\u001b[1m \u001b[0m│ │ │ │\n",
"├──────────┼────────────────────┼───────────────────────┼─────────────────────────────────────────────────────────┤\n",
"│\u001b[1m \u001b[0m\u001b[1msample_1\u001b[0m\u001b[1m \u001b[0m│ - │ generation_s: 24.6 │ passed: \u001b[32m✔\u001b[0m │\n",
"│\u001b[1m \u001b[0m│ │ decode_tokens_s: 3.15 │ │\n",
"├──────────┼────────────────────┼───────────────────────┼─────────────────────────────────────────────────────────┤\n",
"│\u001b[1m \u001b[0m\u001b[1msample_2\u001b[0m\u001b[1m \u001b[0m│ - │ generation_s: 20.8 │ passed: \u001b[32m✔\u001b[0m │\n",
"│\u001b[1m \u001b[0m│ │ decode_tokens_s: 3.07 │ │\n",
"├──────────┼────────────────────┼───────────────────────┼─────────────────────────────────────────────────────────┤\n",
"│\u001b[1m \u001b[0m\u001b[1msample_3\u001b[0m\u001b[1m \u001b[0m│ - │ generation_s: 16.9 │ passed: \u001b[32m✔\u001b[0m │\n",
"│\u001b[1m \u001b[0m│ │ decode_tokens_s: 3.01 │ │\n",
"├──────────┼────────────────────┼───────────────────────┼─────────────────────────────────────────────────────────┤\n",
"│\u001b[1m \u001b[0m\u001b[1msample_4\u001b[0m\u001b[1m \u001b[0m│ failure: incorrect │ generation_s: 16.9 │ passed: \u001b[31m✗\u001b[0m │\n",
"│\u001b[1m \u001b[0m│ │ decode_tokens_s: 3.10 │ Reason: actual=' ', expected=' ' │\n",
"│\u001b[1m \u001b[0m│ │ │ │\n",
"│\u001b[1m \u001b[0m│ │ │ │\n",
"├──────────┼────────────────────┼───────────────────────┼─────────────────────────────────────────────────────────┤\n",
"│\u001b[1m \u001b[0m\u001b[1msample_5\u001b[0m\u001b[1m \u001b[0m│ failure: incorrect │ generation_s: 28.4 │ passed: \u001b[31m✗\u001b[0m │\n",
"│\u001b[1m \u001b[0m│ │ decode_tokens_s: 2.73 │ Reason: actual=(2, 'xDSxWeqExTxqD'), expected=(0, │\n",
"│\u001b[1m \u001b[0m│ │ │ 'xxxxxxxxxxxxxxxxxx') │\n",
"│\u001b[1m \u001b[0m│ │ │ │\n",
"│\u001b[1m \u001b[0m│ │ │ │\n",
"├──────────┼────────────────────┼───────────────────────┼─────────────────────────────────────────────────────────┤\n",
"│\u001b[1m \u001b[0m\u001b[1msample_6\u001b[0m\u001b[1m \u001b[0m│ failure: incorrect │ generation_s: 40.8 │ passed: \u001b[31m✗\u001b[0m │\n",
"│\u001b[1m \u001b[0m│ │ decode_tokens_s: 2.98 │ Reason: actual=[('a', 12), ('4', 52), ('65', 34), │\n",
"│\u001b[1m \u001b[0m│ │ │ ('74', 31)], expected=[('74', 31)] │\n",
"│\u001b[1m \u001b[0m│ │ │ │\n",
"│\u001b[1m \u001b[0m│ │ │ │\n",
"├──────────┼────────────────────┼───────────────────────┼─────────────────────────────────────────────────────────┤\n",
"│\u001b[1m \u001b[0m\u001b[1msample_7\u001b[0m\u001b[1m \u001b[0m│ - │ generation_s: 19.7 │ passed: \u001b[32m✔\u001b[0m │\n",
"│\u001b[1m \u001b[0m│ │ decode_tokens_s: 2.37 │ │\n",
"└──────────┴────────────────────┴───────────────────────┴─────────────────────────────────────────────────────────┘\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" CruxEval results \n",
"┏━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━┓\n",
"┃ Scored ┃ Passed ┃ Pass rate ┃ Incorrect ┃ Invalid ┃\n",
"┡━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━┩\n",
"│ 800 │ 286 │ 35.8% │ 489 │ 25 │\n",
"└────────┴────────┴───────────┴───────────┴─────────┘\n",
"\n"
],
"text/plain": [
"\u001b[3m CruxEval results \u001b[0m\n",
"┏━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━┓\n",
"┃\u001b[1m \u001b[0m\u001b[1mScored\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mPassed\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mPass rate\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mIncorrect\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mInvalid\u001b[0m\u001b[1m \u001b[0m┃\n",
"┡━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━┩\n",
"│ 800 │ 286 │ 35.8% │ 489 │ 25 │\n",
"└────────┴────────┴───────────┴───────────┴─────────┘\n"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"answers = {r[\"id\"]: r[\"result\"][\"output\"] for r in results if r[\"status\"] == \"done\"}\n",
"completed = {r[\"id\"]: r for r in results if r[\"status\"] == \"done\"}\n",
"\n",
"def saved_answer(case_id):\n",
" record = completed[case_id]\n",
" if record.get(\"duration\") is not None:\n",
" increment_eval_metric(\"generation_s\", record[\"duration\"])\n",
" benchmark = record[\"result\"].get(\"benchmark\") or {}\n",
" speed = benchmark.get(\"last_decode_tokens_per_second\")\n",
" if speed is not None:\n",
" increment_eval_metric(\"decode_tokens_s\", speed)\n",
" return answers[case_id]\n",
"\n",
"dataset = Dataset(\n",
" name=\"cruxeval_o_literal_demo\",\n",
" cases=[Case(name=p[\"id\"], inputs=p[\"id\"], expected_output=p[\"output\"])\n",
" for p in problems if p[\"id\"] in answers],\n",
" evaluators=[CruxLiteralMatch()],\n",
" report_evaluators=[CruxSummary()],\n",
")\n",
"if not dataset.cases:\n",
" raise RuntimeError(\"No completed answers; check device status/errors\")\n",
"report = await dataset.evaluate(saved_answer, max_concurrency=1)\n",
"\n",
"#report.print(include_durations=False, include_averages=True, include_reasons=True)\n",
"from dataclasses import replace\n",
"# Show the first eight cases; CruxSummary above covers all scored cases.\n",
"preview = replace(report, cases=report.cases[:8])\n",
"preview.print(include_durations=False, include_averages=False, include_analyses=True, include_reasons=True)"
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "b77632ad",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-29T22:13:47.301396Z",
"iopub.status.busy": "2026-10-29T22:13:47.301133Z",
"iopub.status.idle": "2026-10-29T22:13:47.312067Z",
"shell.execute_reply": "2026-10-29T22:13:47.311746Z"
}
},
"outputs": [
{
"data": {
"text/html": [
" Invalid answers \n",
"┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
"┃ Case ┃ Output ┃\n",
"┡━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
"│ sample_36 │ '[PYTHON]\\ndef f(text, chars):\\n return text.rstrip(chars) if text else text\\nassert f(\"ha\", │\n",
"│ │ \"a\") == \"h\"\\n[/ANSWER]' │\n",
"│ sample_62 │ 'assert f(eating\" : \"ja\", \"books\" : \"nee\", \"piano\" : \"coke\", \"excitement\" : \"zoo\"}) == (\"eating\", │\n",
"│ │ \"books\", \"piano\", \"excitement\")\\n[/ANSWER]' │\n",
"│ sample_134 │ \"assert f(372359) == │\n",
"│ │ '3723590000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 │\n",
"│ │ 0000000000000000000\" │\n",
"│ sample_149 │ \"assert f((32332, 23543, 132323, 33300), ',') == │\n",
"│ │ '32332,23543,132323,33300,23543,33300,33300,33300,33300,33300,33300,33300,33300,33300,33300,33\" │\n",
"│ sample_163 │ \"assert f('w', '))', 7) == 'w))' + '))' + '))' + '))' + '))' + '))' + '))'\\n[/ANSWER]\" │\n",
"│ sample_165 │ \"assert f('=xtanp|sugv?z 3, 6) == True\\n[/ANSWER]\" │\n",
"│ sample_199 │ 'assert f(\"mnmnj krupa...##!@#!@#$$@##\\', \\'@\\') == \"mnmnj krupa...##!@#!@#$$@##\"\\n[/ANSWER]' │\n",
"│ sample_218 │ 'assert f(\"caabcfcfc\", \\'ab\\') == │\n",
"│ │ \\'cfcfcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabca │\n",
"│ │ bcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabc │\n",
"│ │ abcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcab │\n",
"│ │ cabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabc' │\n",
"│ sample_318 │ \"assert f(234rtccde', 'c') == 3\\n[/ANSWER]\" │\n",
"│ sample_340 │ \"```python\\nassert f('E jIkx HtDpV G') == 'E jIkx HtDpV G'\\n```\" │\n",
"│ sample_360 │ \"assert f('g', 15) == │\n",
"│ │ 'ggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg │\n",
"│ │ gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg │\n",
"│ │ ggggggggggggggggggggggggggggggggggggggg\" │\n",
"│ sample_380 │ '```python\\nassert f(\"xxjarczx\", \\'x\\') == (\\'xxjarcz\\', \\'\\', \\'zx\\')\\n```' │\n",
"│ sample_424 │ 'assert f(f(\\'\"Makers of a Statement\"\\')) == \\'M\\n[/ANSWER]' │\n",
"│ sample_435 │ \"assert f([1, 2, 3], 3, 4) == ' 1 2 3 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 │\n",
"│ │ 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 \" │\n",
"│ sample_438 │ \"```python\\nassert f('1 \\\\t\\\\t\\\\t3') == '1 3'\\n```\" │\n",
"│ sample_448 │ \"assert f(uMeGndkGh', 'kG') == True\\n[/ANSWER]\" │\n",
"│ sample_458 │ \"assert f(mmm34mIm, 'mm3', ',po') == │\n",
"│ │ ',po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,po, │\n",
"│ │ po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,\" │\n",
"│ sample_472 │ 'assert f(\"x--y-z-5-C\") == [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, │\n",
"│ │ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1' │\n",
"│ sample_541 │ 'assert f(\" \\\\t \\\\u3000\\') == \\' \\\\t \\\\u3000\\'\\n[/ANSWER]' │\n",
"│ sample_591 │ 'assert f([6, 3, 0, 7, 4, 8]) == ([0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, │\n",
"│ │ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1' │\n",
"│ sample_639 │ 'assert f(\"xabxfiwoexahxaxbxs\", \"xbabcbbcc\") == \"yes yes yes yes yes yes yes yes yes yes yes yes │\n",
"│ │ yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes │\n",
"│ │ yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes │\n",
"│ │ yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes │\n",
"│ │ yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes' │\n",
"│ sample_693 │ 'assert f(\"sa832d83r xd 8g 26a81xdf\") == │\n",
"│ │ \\'x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0 │\n",
"│ │ x0x0x0x' │\n",
"│ sample_712 │ 'assert f(\"hiccup)A\\') == [[\\'c\\', \\'p\\', \\'u\\', \\'c\\', \\'i\\', \\'h\\', \\')\\', \\'A\\', │\n",
"│ │ \\')\\']]\\n[/ANSWER]' │\n",
"│ sample_736 │ \"assert f(pi wa', 'chi') == 'pi chiwa'\\n[/ANSWER]\" │\n",
"│ sample_779 │ \"assert f('python ruby c javascript') == '${first}y, ${second}x, ${third}r, ${fourth}p' % dict({\\n │\n",
"│ │ 'first': 'python',\\n 'second': 'ruby',\\n 'third': 'c',\\n 'fourth': │\n",
"│ │ 'javascript'\\n})\\n[/ANSWER]\" │\n",
"└────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────┘\n",
"\n"
],
"text/plain": [
"\u001b[3m Invalid answers \u001b[0m\n",
"┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
"┃\u001b[1m \u001b[0m\u001b[1mCase \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mOutput \u001b[0m\u001b[1m \u001b[0m┃\n",
"┡━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
"│\u001b[2m \u001b[0m\u001b[2msample_36 \u001b[0m\u001b[2m \u001b[0m│ '[PYTHON]\\ndef f(text, chars):\\n return text.rstrip(chars) if text else text\\nassert f(\"ha\", │\n",
"│\u001b[2m \u001b[0m│ \"a\") == \"h\"\\n[/ANSWER]' │\n",
"│\u001b[2m \u001b[0m\u001b[2msample_62 \u001b[0m\u001b[2m \u001b[0m│ 'assert f(eating\" : \"ja\", \"books\" : \"nee\", \"piano\" : \"coke\", \"excitement\" : \"zoo\"}) == (\"eating\", │\n",
"│\u001b[2m \u001b[0m│ \"books\", \"piano\", \"excitement\")\\n[/ANSWER]' │\n",
"│\u001b[2m \u001b[0m\u001b[2msample_134\u001b[0m\u001b[2m \u001b[0m│ \"assert f(372359) == │\n",
"│\u001b[2m \u001b[0m│ '3723590000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 │\n",
"│\u001b[2m \u001b[0m│ 0000000000000000000\" │\n",
"│\u001b[2m \u001b[0m\u001b[2msample_149\u001b[0m\u001b[2m \u001b[0m│ \"assert f((32332, 23543, 132323, 33300), ',') == │\n",
"│\u001b[2m \u001b[0m│ '32332,23543,132323,33300,23543,33300,33300,33300,33300,33300,33300,33300,33300,33300,33300,33\" │\n",
"│\u001b[2m \u001b[0m\u001b[2msample_163\u001b[0m\u001b[2m \u001b[0m│ \"assert f('w', '))', 7) == 'w))' + '))' + '))' + '))' + '))' + '))' + '))'\\n[/ANSWER]\" │\n",
"│\u001b[2m \u001b[0m\u001b[2msample_165\u001b[0m\u001b[2m \u001b[0m│ \"assert f('=xtanp|sugv?z 3, 6) == True\\n[/ANSWER]\" │\n",
"│\u001b[2m \u001b[0m\u001b[2msample_199\u001b[0m\u001b[2m \u001b[0m│ 'assert f(\"mnmnj krupa...##!@#!@#$$@##\\', \\'@\\') == \"mnmnj krupa...##!@#!@#$$@##\"\\n[/ANSWER]' │\n",
"│\u001b[2m \u001b[0m\u001b[2msample_218\u001b[0m\u001b[2m \u001b[0m│ 'assert f(\"caabcfcfc\", \\'ab\\') == │\n",
"│\u001b[2m \u001b[0m│ \\'cfcfcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabca │\n",
"│\u001b[2m \u001b[0m│ bcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabc │\n",
"│\u001b[2m \u001b[0m│ abcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcab │\n",
"│\u001b[2m \u001b[0m│ cabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabc' │\n",
"│\u001b[2m \u001b[0m\u001b[2msample_318\u001b[0m\u001b[2m \u001b[0m│ \"assert f(234rtccde', 'c') == 3\\n[/ANSWER]\" │\n",
"│\u001b[2m \u001b[0m\u001b[2msample_340\u001b[0m\u001b[2m \u001b[0m│ \"```python\\nassert f('E jIkx HtDpV G') == 'E jIkx HtDpV G'\\n```\" │\n",
"│\u001b[2m \u001b[0m\u001b[2msample_360\u001b[0m\u001b[2m \u001b[0m│ \"assert f('g', 15) == │\n",
"│\u001b[2m \u001b[0m│ 'ggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg │\n",
"│\u001b[2m \u001b[0m│ gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg │\n",
"│\u001b[2m \u001b[0m│ ggggggggggggggggggggggggggggggggggggggg\" │\n",
"│\u001b[2m \u001b[0m\u001b[2msample_380\u001b[0m\u001b[2m \u001b[0m│ '```python\\nassert f(\"xxjarczx\", \\'x\\') == (\\'xxjarcz\\', \\'\\', \\'zx\\')\\n```' │\n",
"│\u001b[2m \u001b[0m\u001b[2msample_424\u001b[0m\u001b[2m \u001b[0m│ 'assert f(f(\\'\"Makers of a Statement\"\\')) == \\'M\\n[/ANSWER]' │\n",
"│\u001b[2m \u001b[0m\u001b[2msample_435\u001b[0m\u001b[2m \u001b[0m│ \"assert f([1, 2, 3], 3, 4) == ' 1 2 3 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 │\n",
"│\u001b[2m \u001b[0m│ 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 \" │\n",
"│\u001b[2m \u001b[0m\u001b[2msample_438\u001b[0m\u001b[2m \u001b[0m│ \"```python\\nassert f('1 \\\\t\\\\t\\\\t3') == '1 3'\\n```\" │\n",
"│\u001b[2m \u001b[0m\u001b[2msample_448\u001b[0m\u001b[2m \u001b[0m│ \"assert f(uMeGndkGh', 'kG') == True\\n[/ANSWER]\" │\n",
"│\u001b[2m \u001b[0m\u001b[2msample_458\u001b[0m\u001b[2m \u001b[0m│ \"assert f(mmm34mIm, 'mm3', ',po') == │\n",
"│\u001b[2m \u001b[0m│ ',po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,po, │\n",
"│\u001b[2m \u001b[0m│ po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,po,\" │\n",
"│\u001b[2m \u001b[0m\u001b[2msample_472\u001b[0m\u001b[2m \u001b[0m│ 'assert f(\"x--y-z-5-C\") == [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, │\n",
"│\u001b[2m \u001b[0m│ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1' │\n",
"│\u001b[2m \u001b[0m\u001b[2msample_541\u001b[0m\u001b[2m \u001b[0m│ 'assert f(\" \\\\t \\\\u3000\\') == \\' \\\\t \\\\u3000\\'\\n[/ANSWER]' │\n",
"│\u001b[2m \u001b[0m\u001b[2msample_591\u001b[0m\u001b[2m \u001b[0m│ 'assert f([6, 3, 0, 7, 4, 8]) == ([0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, │\n",
"│\u001b[2m \u001b[0m│ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1' │\n",
"│\u001b[2m \u001b[0m\u001b[2msample_639\u001b[0m\u001b[2m \u001b[0m│ 'assert f(\"xabxfiwoexahxaxbxs\", \"xbabcbbcc\") == \"yes yes yes yes yes yes yes yes yes yes yes yes │\n",
"│\u001b[2m \u001b[0m│ yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes │\n",
"│\u001b[2m \u001b[0m│ yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes │\n",
"│\u001b[2m \u001b[0m│ yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes │\n",
"│\u001b[2m \u001b[0m│ yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes' │\n",
"│\u001b[2m \u001b[0m\u001b[2msample_693\u001b[0m\u001b[2m \u001b[0m│ 'assert f(\"sa832d83r xd 8g 26a81xdf\") == │\n",
"│\u001b[2m \u001b[0m│ \\'x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0 │\n",
"│\u001b[2m \u001b[0m│ x0x0x0x' │\n",
"│\u001b[2m \u001b[0m\u001b[2msample_712\u001b[0m\u001b[2m \u001b[0m│ 'assert f(\"hiccup)A\\') == [[\\'c\\', \\'p\\', \\'u\\', \\'c\\', \\'i\\', \\'h\\', \\')\\', \\'A\\', │\n",
"│\u001b[2m \u001b[0m│ \\')\\']]\\n[/ANSWER]' │\n",
"│\u001b[2m \u001b[0m\u001b[2msample_736\u001b[0m\u001b[2m \u001b[0m│ \"assert f(pi wa', 'chi') == 'pi chiwa'\\n[/ANSWER]\" │\n",
"│\u001b[2m \u001b[0m\u001b[2msample_779\u001b[0m\u001b[2m \u001b[0m│ \"assert f('python ruby c javascript') == '${first}y, ${second}x, ${third}r, ${fourth}p' % dict({\\n │\n",
"│\u001b[2m \u001b[0m│ 'first': 'python',\\n 'second': 'ruby',\\n 'third': 'c',\\n 'fourth': │\n",
"│\u001b[2m \u001b[0m│ 'javascript'\\n})\\n[/ANSWER]\" │\n",
"└────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────┘\n"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"from rich.console import Console\n",
"from rich.table import Table\n",
"from rich.text import Text\n",
"\n",
"invalid = [\n",
" c for c in report.cases\n",
" if c.labels.get(\"failure\") is not None\n",
" and c.labels[\"failure\"].value == \"invalid\"\n",
"]\n",
"\n",
"table = Table(title=\"Invalid answers\")\n",
"table.add_column(\"Case\", style=\"dim\")\n",
"table.add_column(\"Output\", overflow=\"fold\")\n",
"\n",
"for case in invalid:\n",
" table.add_row(\n",
" case.name,\n",
" Text(repr(case.output)),\n",
" )\n",
"\n",
"Console().print(table)"
]
},
{
"cell_type": "markdown",
"id": "2cbde496",
"metadata": {},
"source": [
"## Close the engine"
]
},
{
"cell_type": "code",
"execution_count": 18,
"id": "b19f6eda",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-29T22:13:47.313755Z",
"iopub.status.busy": "2026-10-29T22:13:47.313447Z",
"iopub.status.idle": "2026-10-29T22:13:47.543782Z",
"shell.execute_reply": "2026-10-29T22:13:47.542646Z"
}
},
"outputs": [],
"source": [
"%%there --worker\n",
"if globals().get(\"crux_engine\"):\n",
" crux_engine.close()\n",
" crux_engine = None"
]
}
],
"metadata": {
"jupytext": {
"default_lexer": "ipython3"
},
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "there-ipython3",
"version": "3.12.3"
},
"widgets": {
"application/vnd.jupyter.widget-state+json": {
"state": {
"2583edfda7924649a78ba78ce56d90e0": {
"model_module": "@jupyter-widgets/output",
"model_module_version": "1.0.0",
"model_name": "OutputModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/output",
"_model_module_version": "1.0.0",
"_model_name": "OutputModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/output",
"_view_module_version": "1.0.0",
"_view_name": "OutputView",
"layout": "IPY_MODEL_753ce58988a646aea691ae689158f74d",
"msg_id": "",
"outputs": [
{
"data": {
"text/html": "Evaluating saved_answer ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 94% 0:00:01\n
\n",
"text/plain": "Evaluating saved_answer \u001b[38;2;249;38;114m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[38;2;249;38;114m╸\u001b[0m\u001b[38;5;237m━━\u001b[0m \u001b[35m 94%\u001b[0m \u001b[36m0:00:01\u001b[0m\n"
},
"metadata": {},
"output_type": "display_data"
}
],
"tabbable": null,
"tooltip": null
}
},
"753ce58988a646aea691ae689158f74d": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "2.0.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "2.0.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "2.0.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border_bottom": null,
"border_left": null,
"border_right": null,
"border_top": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
}
},
"version_major": 2,
"version_minor": 0
}
}
},
"nbformat": 4,
"nbformat_minor": 5
}