{ "cells": [ { "cell_type": "markdown", "id": "f47250fa", "metadata": {}, "source": [ "# LeetCode\n", "\n", "Companion notebook for the video [*Benchmarking Local AI on Android with PythonHere and Pydantic Evals*](https://youtu.be/CSwZkyGWrhU).\n", "\n", "Generate Python solutions on Android using\n", "[newfacade/LeetCodeDataset](https://huggingface.co/datasets/newfacade/LeetCodeDataset),\n", "then run the dataset's tests locally with\n", "[Pydantic Evals](https://pydantic.dev/docs/ai/evals/evals/).\n", "\n", "## Connect and configure" ] }, { "cell_type": "code", "execution_count": 1, "id": "8c7f0d1a", "metadata": { "execution": { "iopub.execute_input": "2026-10-29T22:54:23.837715Z", "iopub.status.busy": "2026-10-29T22:54:23.837448Z", "iopub.status.idle": "2026-10-29T22:54:23.839908Z", "shell.execute_reply": "2026-10-29T22:54:23.839621Z" } }, "outputs": [], "source": [ "# %pip install pydantic-evals datasets" ] }, { "cell_type": "code", "execution_count": 2, "id": "a766c8ef", "metadata": { "execution": { "iopub.execute_input": "2026-10-29T22:54:23.841550Z", "iopub.status.busy": "2026-10-29T22:54:23.841261Z", "iopub.status.idle": "2026-10-29T22:54:24.428502Z", "shell.execute_reply": "2026-10-29T22:54:24.428080Z" } }, "outputs": [], "source": [ "%load_ext pythonhere\n", "%connect-there" ] }, { "cell_type": "markdown", "id": "6a6b2188", "metadata": {}, "source": [ "We use Qwen2.5-Coder-3B-Instruct with greedy decoding (`top_k=1`), seed 42,\n", "and a 1,000-token answer limit. `%there put` copies the run name and settings\n", "to the device.\n", "\n", "`LIMIT=None` selects the full test split; set it to `3` for a short run.\n", "Reuse `RUN_NAME` to resume, or choose a new name when changing settings or\n", "selected cases. `REVISION` pins the dataset version." ] }, { "cell_type": "code", "execution_count": 3, "id": "f9dbb57b", "metadata": { "execution": { "iopub.execute_input": "2026-10-29T22:54:24.430731Z", "iopub.status.busy": "2026-10-29T22:54:24.430529Z", "iopub.status.idle": "2026-10-29T22:54:24.629034Z", "shell.execute_reply": "2026-10-29T22:54:24.628656Z" } }, "outputs": [], "source": [ "import os\n", "RUN_NAME = \"leetcode-qwen-greedy-full-001\"\n", "LIMIT = None # All cases.\n", "REVISION = \"215604aeed660029df7de2fea5a4d7b6ed476a08\"\n", "settings = {\n", " \"model_repo\": \"litert-community/Qwen2.5-Coder-3B-Instruct\",\n", " \"model_file\": \"Qwen2.5_Coder_3B_It.litertlm\",\n", " \"sampler\": {\"temperature\": 1.0, \"top_p\": 1.0, \"top_k\": 1, \"seed\": 42},\n", " \"max_output_tokens\": 1000,\n", " \"dataset_revision\": REVISION,\n", "}\n", "hf_token = os.environ.get(\"HF_TOKEN\")\n", "\n", "%there put RUN_NAME\n", "%there put settings\n", "%there put hf_token" ] }, { "cell_type": "markdown", "id": "9489aa76", "metadata": {}, "source": [ "## Download the model" ] }, { "cell_type": "code", "execution_count": 4, "id": "6fc647d1", "metadata": { "execution": { "iopub.execute_input": "2026-10-29T22:54:24.630743Z", "iopub.status.busy": "2026-10-29T22:54:24.630600Z", "iopub.status.idle": "2026-10-29T22:54:25.031048Z", "shell.execute_reply": "2026-10-29T22:54:25.030722Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/storage/emulated/0/Android/data/me.herethere.pythonhere_dev/files/models/litert-community/Qwen2.5-Coder-3B-Instruct/Qwen2.5_Coder_3B_It.litertlm\n" ] } ], "source": [ "%%there --worker\n", "from ml_here import download_hf_model, require_model\n", "\n", "try:\n", " model = require_model(settings[\"model_repo\"], settings[\"model_file\"])\n", "except FileNotFoundError:\n", " def report_progress(value):\n", " if value.percent is not None:\n", " print(f\"\\rDownloading: {value.percent:.1f}%\", end=\"\", flush=True)\n", "\n", " model = download_hf_model(\n", " repo_id=settings[\"model_repo\"],\n", " filename=settings[\"model_file\"],\n", " token=hf_token,\n", " progress=report_progress,\n", " )\n", "\n", "print(model)" ] }, { "cell_type": "markdown", "id": "363f0a2b", "metadata": {}, "source": [ "## Load dataset and build prompts\n", "\n", "We send each problem's original `query` to the model. Tests stay local and\n", "are used only for scoring. Inspect one prompt and its tests below." ] }, { "cell_type": "code", "execution_count": 5, "id": "5e08970e", "metadata": { "execution": { "iopub.execute_input": "2026-10-29T22:54:25.032742Z", "iopub.status.busy": "2026-10-29T22:54:25.032491Z", "iopub.status.idle": "2026-10-29T22:54:28.360039Z", "shell.execute_reply": "2026-10-29T22:54:28.359589Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "'Selected 228 cases'\n" ] } ], "source": [ "from datasets import load_dataset\n", "from pprint import pprint as pp\n", "\n", "rows = load_dataset(\"newfacade/LeetCodeDataset\", revision=REVISION, split=\"test\")\n", "if LIMIT is not None:\n", " rows = rows.select(range(LIMIT))\n", "problems = [dict(row) for row in rows]\n", "assert problems, \"No cases selected\"\n", "assert len({p[\"task_id\"] for p in problems}) == len(problems)\n", "pp(f\"Selected {len(problems)} cases\")" ] }, { "cell_type": "code", "execution_count": 6, "id": "34fcff29", "metadata": { "execution": { "iopub.execute_input": "2026-10-29T22:54:28.361838Z", "iopub.status.busy": "2026-10-29T22:54:28.361487Z", "iopub.status.idle": "2026-10-29T22:54:28.363772Z", "shell.execute_reply": "2026-10-29T22:54:28.363456Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "You are an expert Python programmer. You will be given a question (problem specification) and will generate a correct Python program that matches the specification and passes all tests.\n", "\n", "### Question:\n", "You are given an integer n and a 2D integer array queries.\n", "There are n cities numbered from 0 to n - 1. Initially, there is a unidirectional road from city i to city i + 1 for all 0 <= i < n - 1.\n", "queries[i] = [ui, vi] represents the addition of a new unidirectional road from city ui to city vi. After each query, you need to find the length of the shortest path from city 0 to city n - 1.\n", "Return an array answer where for each i in the range [0, queries.length - 1], answer[i] is the length of the shortest path from city 0 to city n - 1 after processing the first i + 1 queries.\n", " \n", "Example 1:\n", "\n", "Input: n = 5, queries = [[2,4],[0,2],[0,4]]\n", "Output: [3,2,1]\n", "Explanation: \n", "\n", "After the addition of the road from 2 to 4, the length of the shortest path from 0 to 4 is 3.\n", "\n", "After the addition of the road from 0 to 2, the length of the shortest path from 0 to 4 is 2.\n", "\n", "After the addition of the road from 0 to 4, the length of the shortest path from 0 to 4 is 1.\n", "\n", "Example 2:\n", "\n", "Input: n = 4, queries = [[0,3],[0,2]]\n", "Output: [1,1]\n", "Explanation:\n", "\n", "After the addition of the road from 0 to 3, the length of the shortest path from 0 to 3 is 1.\n", "\n", "After the addition of the road from 0 to 2, the length of the shortest path remains 1.\n", "\n", " \n", "Constraints:\n", "\n", "3 <= n <= 500\n", "1 <= queries.length <= 500\n", "queries[i].length == 2\n", "0 <= queries[i][0] < queries[i][1] < n\n", "1 < queries[i][1] - queries[i][0]\n", "There are no repeated roads among the queries.\n", "\n", "\n", "\n", "### Format: You will use the following starter code to write the solution to the problem and enclose your code within delimiters.\n", "```python\n", "class Solution:\n", " def shortestDistanceAfterQueries(self, n: int, queries: List[List[int]]) -> List[int]:\n", " \n", "```\n", "\n", "### Answer: (use the provided format with backticks)\n", "\n" ] } ], "source": [ "print(problems[0][\"query\"])" ] }, { "cell_type": "code", "execution_count": 7, "id": "e2b17a07", "metadata": { "execution": { "iopub.execute_input": "2026-10-29T22:54:28.365342Z", "iopub.status.busy": "2026-10-29T22:54:28.365103Z", "iopub.status.idle": "2026-10-29T22:54:28.367474Z", "shell.execute_reply": "2026-10-29T22:54:28.367061Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "def check(candidate):\n", " assert candidate(n = 7,queries = [[0, 5], [1, 6], [2, 4]]) == [2, 2, 2]\n", " assert candidate(n = 8,queries = [[1, 5], [2, 6], [3, 7], [0, 4], [0, 6], [0, 7]]) == [4, 4, 4, 4, 2, 1]\n", " assert candidate(n = 6,queries = [[1, 3], [2, 5], [0, 3]]) == [4, 3, 3]\n", " assert candidate(n = 8,queries = [[0, 6], [1, 7], [2, 5], [3, 4]]) == [2, 2, 2, 2]\n", " assert candidate(n = 7,queries = [[1, 5], [2, 6], [3, 5]]) == [3, 3, 3]\n", " assert candidate(n = 6,queries = [[1, 3], [2, 4], [0, 3]]) == [4, 4, 3]\n", " assert candidate(n = 6,queries = [[1, 3], [2, 5], [0, 1], [3, 4]]) == [4, 3, 3, 3]\n", " assert candidate(n = 5,queries = [[2, 4], [0, 2], [0, 4]]) == [3, 2, 1]\n", " assert candidate(n = 4,queries = [[0, 3], [0, 2]]) == [1, 1]\n", " assert candidate(n = 7,queries = [[1, 4], [2, 6], [3, 5], [0, 3], [0, 5]]) == [4, 3, 3, 3, 2]\n", " assert candidate(n = 50,queries = [[0, 49], [0, 48], [1, 47], [2, 46], [3, 45], [4, 44], [5, 43], [6, 42], [7, 41], [8, 40], [9, 39], [10, 38], [11, 37], [12, 36], [13, 35], [14, 34], [15, 33], [16, 32], [17, 31], [18, 30], [19, 29], [20, 28], [21, 27], [22, 26], [23, 25]]) == [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\n", " assert candidate(n = 50,queries = [[0, 25], [0, 30], [0, 35], [0, 40], [0, 45], [5, 15], [5, 20], [5, 25], [5, 30], [5, 35], [5, 40], [5, 45], [10, 20], [10, 25], [10, 30], [10, 35], [10, 40], [10, 45], [15, 25], [15, 30], [15, 35], [15, 40], [15, 45], [20, 30], [20, 35], [20, 40], [20, 45], [25, 35], [25, 40], [25, 45], [30, 40], [30, 45], [35, 45]]) == [25, 20, 15, 10, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5]\n", " assert candidate(n = 45,queries = [[0, 18], [5, 28], [10, 34], [15, 39], [20, 41], [25, 43], [30, 44], [35, 42], [1, 19], [6, 30], [11, 35], [16, 40], [21, 42], [26, 36], [31, 37]]) == [27, 22, 21, 21, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7]\n", " assert candidate(n = 15,queries = [[0, 10], [1, 8], [2, 12], [3, 7], [4, 9], [5, 11], [6, 13], [7, 14]]) == [5, 5, 5, 5, 5, 5, 5, 5]\n", " assert candidate(n = 12,queries = [[0, 5], [0, 11], [1, 10], [2, 8], [3, 9], [4, 7], [5, 6]]) == [7, 1, 1, 1, 1, 1, 1]\n", " assert candidate(n = 20,queries = [[0, 19], [0, 10], [1, 18], [2, 17], [3, 16], [4, 15], [5, 14], [6, 13], [7, 12], [8, 9]]) == [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\n", " assert candidate(n = 10,queries = [[0, 9], [1, 8], [2, 7], [3, 6], [4, 5], [0, 7], [2, 5], [1, 4]]) == [1, 1, 1, 1, 1, 1, 1, 1]\n", " assert candidate(n = 10,queries = [[0, 5], [1, 4], [2, 6], [3, 7], [8, 9]]) == [5, 5, 5, 5, 5]\n", " assert candidate(n = 50,queries = [[0, 49], [1, 48], [2, 47], [3, 46], [4, 45], [5, 44], [6, 43], [7, 42], [8, 41], [9, 40], [10, 39], [11, 38], [12, 37], [13, 36], [14, 35], [15, 34], [16, 33], [17, 32], [18, 31], [19, 30], [0, 47], [2, 45], [4, 43], [6, 41], [8, 39], [10, 38], [12, 37], [14, 36], [16, 35], [18, 34], [20, 33], [22, 32], [24, 31], [26, 30], [28, 29]]) == [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\n", " assert candidate(n = 20,queries = [[0, 10], [1, 15], [2, 19], [3, 18], [4, 17], [5, 16], [6, 15], [7, 14], [8, 13], [9, 12], [10, 19], [11, 18], [12, 17], [13, 16], [14, 15]]) == [10, 6, 3, 3, 3, 3, 3, 3, 3, 3, 2, 2, 2, 2, 2]\n", " assert candidate(n = 12,queries = [[0, 11], [0, 6], [1, 10], [2, 9], [3, 8], [4, 7], [5, 6]]) == [1, 1, 1, 1, 1, 1, 1]\n", " assert candidate(n = 50,queries = [[0, 49], [1, 48], [2, 47], [3, 46], [4, 45], [5, 44], [6, 43], [7, 42], [8, 41], [9, 40], [10, 39], [11, 38], [12, 37], [13, 36], [14, 35], [15, 34], [16, 33], [17, 32], [18, 31], [19, 30], [20, 29], [21, 28], [22, 27], [23, 26], [24, 25], [25, 24], [26, 23], [27, 22], [28, 21], [29, 20], [30, 19], [31, 18], [32, 17], [33, 16], [34, 15], [35, 14], [36, 13], [37, 12], [38, 11], [39, 10], [40, 9], [41, 8], [42, 7], [43, 6], [44, 5], [45, 4], [46, 3], [47, 2], [48, 1]]) == [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\n", " assert candidate(n = 30,queries = [[0, 10], [0, 15], [0, 20], [5, 25], [5, 29], [10, 19], [10, 24], [15, 23], [15, 28], [20, 22], [20, 27], [25, 26]]) == [20, 15, 10, 10, 6, 6, 6, 6, 3, 3, 3, 3]\n", " assert candidate(n = 100,queries = [[0, 99], [0, 98], [1, 97], [2, 96], [3, 95], [4, 94], [5, 93], [6, 92], [7, 91], [8, 90], [9, 89], [10, 88], [11, 87], [12, 86], [13, 85], [14, 84], [15, 83], [16, 82], [17, 81], [18, 80], [19, 79], [20, 78], [21, 77], [22, 76], [23, 75], [24, 74], [25, 73], [26, 72], [27, 71], [28, 70], [29, 69], [30, 68], [31, 67], [32, 66], [33, 65], [34, 64], [35, 63], [36, 62], [37, 61], [38, 60], [39, 59], [40, 58], [41, 57], [42, 56], [43, 55], [44, 54], [45, 53], [46, 52], [47, 51], [48, 50]]) == [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\n", " assert candidate(n = 30,queries = [[0, 20], [1, 21], [2, 22], [3, 23], [4, 24], [5, 25], [6, 26], [7, 27], [8, 28], [9, 29], [10, 25], [11, 26], [12, 27], [13, 28], [14, 29], [15, 28], [16, 29], [0, 15], [2, 21], [4, 23], [6, 25], [8, 27]]) == [10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 3, 3, 3, 3, 3]\n", " assert candidate(n = 15,queries = [[0, 10], [1, 11], [2, 12], [3, 13], [4, 14], [5, 13], [6, 14], [0, 9], [2, 11], [4, 12]]) == [5, 5, 5, 5, 5, 5, 5, 5, 5, 5]\n", " assert candidate(n = 15,queries = [[0, 14], [1, 13], [2, 12], [3, 11], [4, 10], [5, 9], [6, 8], [0, 12], [1, 10], [2, 9], [3, 8], [4, 7], [5, 6]]) == [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\n", " assert candidate(n = 20,queries = [[0, 19], [1, 18], [2, 17], [3, 16], [4, 15], [5, 14], [6, 13], [7, 12], [8, 11], [9, 10], [0, 17], [2, 15], [4, 13], [6, 11], [8, 9]]) == [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\n", " assert candidate(n = 12,queries = [[0, 6], [2, 8], [3, 9], [5, 10], [7, 11], [1, 5]]) == [6, 6, 6, 6, 3, 3]\n", " assert candidate(n = 10,queries = [[0, 9], [1, 8], [2, 7], [3, 6], [4, 5]]) == [1, 1, 1, 1, 1]\n", " assert candidate(n = 40,queries = [[0, 15], [5, 25], [10, 30], [15, 35], [20, 37], [25, 38], [30, 39], [1, 16], [6, 26], [11, 31], [16, 36], [21, 32], [26, 33]]) == [25, 20, 20, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6]\n", " assert candidate(n = 10,queries = [[0, 7], [2, 9], [3, 6], [0, 5], [4, 8]]) == [3, 3, 3, 3, 3]\n", " assert candidate(n = 15,queries = [[0, 14], [1, 10], [2, 8], [3, 9], [4, 11], [5, 12], [6, 13]]) == [1, 1, 1, 1, 1, 1, 1]\n", " assert candidate(n = 20,queries = [[0, 10], [1, 11], [2, 12], [3, 13], [4, 14], [5, 15], [6, 16], [7, 17], [8, 18], [9, 19], [0, 15], [1, 16]]) == [10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 5, 5]\n", " assert candidate(n = 12,queries = [[0, 2], [0, 4], [0, 6], [0, 8], [0, 10], [1, 3], [1, 5], [1, 7], [1, 9], [1, 11]]) == [10, 8, 6, 4, 2, 2, 2, 2, 2, 2]\n", " assert candidate(n = 15,queries = [[0, 7], [1, 9], [2, 11], [3, 10], [4, 12], [5, 13], [6, 14]]) == [8, 7, 6, 6, 6, 6, 6]\n", " assert candidate(n = 10,queries = [[0, 5], [1, 7], [3, 8], [2, 6], [0, 9]]) == [5, 4, 4, 4, 1]\n", " assert candidate(n = 15,queries = [[0, 14], [0, 7], [1, 12], [2, 11], [3, 10], [4, 9], [5, 8], [6, 7]]) == [1, 1, 1, 1, 1, 1, 1, 1]\n", " assert candidate(n = 30,queries = [[0, 25], [1, 26], [2, 27], [3, 28], [4, 29], [5, 15], [6, 16], [7, 17], [8, 18], [9, 19], [10, 20], [11, 21], [12, 22], [13, 23], [14, 24]]) == [5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5]\n", " assert candidate(n = 10,queries = [[0, 5], [1, 6], [2, 7], [3, 8], [4, 9], [0, 7], [2, 9]]) == [5, 5, 5, 5, 5, 3, 3]\n", " assert candidate(n = 25,queries = [[0, 15], [1, 16], [2, 17], [3, 18], [4, 19], [5, 20], [6, 21], [7, 22], [8, 23], [9, 24], [10, 20], [11, 21], [12, 22], [13, 23], [14, 24]]) == [10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10]\n", " assert candidate(n = 30,queries = [[0, 12], [5, 18], [10, 22], [15, 24], [20, 28], [25, 29], [1, 13], [6, 19], [11, 23], [16, 25]]) == [18, 17, 17, 10, 10, 7, 7, 7, 7, 7]\n", " assert candidate(n = 10,queries = [[0, 4], [0, 5], [0, 8], [0, 6], [1, 3], [2, 7], [3, 9], [4, 8]]) == [6, 5, 2, 2, 2, 2, 2, 2]\n", " assert candidate(n = 30,queries = [[0, 29], [1, 28], [2, 27], [3, 26], [4, 25], [5, 24], [6, 23], [7, 22], [8, 21], [9, 20], [10, 19], [11, 18], [12, 17], [13, 16], [14, 15], [15, 14], [16, 13], [17, 12], [18, 11], [19, 10], [20, 9], [21, 8], [22, 7], [23, 6], [24, 5], [25, 4], [26, 3], [27, 2], [28, 1]]) == [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\n", " assert candidate(n = 15,queries = [[0, 14], [1, 13], [2, 12], [3, 11], [4, 10], [5, 9], [6, 8]]) == [1, 1, 1, 1, 1, 1, 1]\n", " assert candidate(n = 10,queries = [[0, 9], [0, 5], [1, 8], [2, 4], [3, 7]]) == [1, 1, 1, 1, 1]\n", " assert candidate(n = 40,queries = [[0, 10], [0, 15], [0, 20], [0, 25], [0, 30], [5, 15], [5, 20], [5, 25], [5, 30], [10, 20], [10, 25], [10, 30], [15, 25], [15, 30], [20, 30]]) == [30, 25, 20, 15, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10]\n", " assert candidate(n = 40,queries = [[0, 35], [1, 36], [2, 37], [3, 38], [4, 39], [5, 25], [6, 26], [7, 27], [8, 28], [9, 29], [10, 30], [11, 31], [12, 32], [13, 33], [14, 34], [15, 20], [16, 21], [17, 22], [18, 23], [19, 24]]) == [5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5]\n", " assert candidate(n = 50,queries = [[0, 45], [1, 46], [2, 47], [3, 48], [4, 49], [5, 35], [6, 36], [7, 37], [8, 38], [9, 39], [10, 40], [11, 41], [12, 42], [13, 43], [14, 44], [15, 30], [16, 31], [17, 32], [18, 33], [19, 34], [20, 25], [21, 26], [22, 27], [23, 28], [24, 29]]) == [5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5]\n", " assert candidate(n = 15,queries = [[0, 14], [1, 12], [2, 13], [3, 11], [4, 9], [5, 10], [6, 8], [7, 10], [8, 13], [9, 14]]) == [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\n", " assert candidate(n = 10,queries = [[0, 5], [5, 9], [2, 6], [1, 8], [3, 7]]) == [5, 2, 2, 2, 2]\n", " assert candidate(n = 20,queries = [[0, 19], [1, 18], [2, 17], [3, 16], [4, 15], [5, 14], [6, 13], [7, 12], [8, 11], [9, 10]]) == [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\n", " assert candidate(n = 20,queries = [[0, 10], [1, 11], [2, 12], [3, 13], [4, 14], [5, 15], [6, 16], [7, 17], [8, 18], [9, 19], [0, 9], [2, 11], [4, 13], [6, 15], [8, 17]]) == [10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 2, 2, 2, 2, 2]\n", " assert candidate(n = 25,queries = [[0, 24], [0, 23], [0, 22], [1, 21], [2, 20], [3, 19], [4, 18], [5, 17], [6, 16], [7, 15], [8, 14], [9, 13], [10, 12]]) == [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\n", " assert candidate(n = 15,queries = [[0, 4], [1, 8], [2, 10], [3, 11], [4, 12], [5, 13], [6, 14], [0, 13], [1, 14]]) == [11, 8, 7, 7, 4, 4, 4, 2, 2]\n", " assert candidate(n = 20,queries = [[0, 15], [5, 18], [10, 19], [1, 12], [3, 17], [7, 16], [9, 14], [2, 13]]) == [5, 5, 5, 5, 5, 5, 5, 5]\n", " assert candidate(n = 10,queries = [[0, 2], [1, 3], [2, 9], [3, 5], [4, 8], [5, 7], [6, 9], [7, 8]]) == [8, 8, 2, 2, 2, 2, 2, 2]\n", " assert candidate(n = 30,queries = [[0, 29], [1, 28], [2, 27], [3, 26], [4, 25], [5, 24], [6, 23], [7, 22], [8, 21], [9, 20], [10, 19], [11, 18], [12, 17], [13, 16], [14, 15], [0, 27], [2, 25], [4, 23], [6, 21], [8, 19], [10, 18], [12, 17], [14, 16]]) == [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\n", " assert candidate(n = 25,queries = [[0, 24], [1, 23], [2, 22], [3, 21], [4, 20], [5, 19], [6, 18], [7, 17], [8, 16], [9, 15], [10, 14], [11, 13], [12, 12], [13, 11], [14, 10], [15, 9], [16, 8], [17, 7], [18, 6], [19, 5], [20, 4], [21, 3], [22, 2], [23, 1]]) == [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\n", " assert candidate(n = 25,queries = [[0, 15], [1, 8], [2, 10], [3, 12], [4, 14], [5, 16], [6, 18], [7, 20], [8, 22], [9, 24], [10, 17], [11, 19], [12, 21], [13, 23]]) == [10, 10, 10, 10, 10, 10, 10, 10, 5, 4, 4, 4, 4, 4]\n", " assert candidate(n = 18,queries = [[0, 17], [0, 9], [1, 16], [2, 15], [3, 14], [4, 13], [5, 12], [6, 11], [7, 10], [8, 9]]) == [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\n", " assert candidate(n = 35,queries = [[0, 13], [5, 20], [10, 24], [15, 28], [20, 32], [25, 34], [30, 33], [1, 14], [6, 21], [11, 25], [16, 29], [21, 31]]) == [22, 20, 20, 10, 9, 9, 8, 8, 8, 8, 8, 8]\n", " assert candidate(n = 15,queries = [[0, 14], [0, 13], [0, 12], [1, 11], [2, 10], [3, 9], [4, 8], [5, 7]]) == [1, 1, 1, 1, 1, 1, 1, 1]\n", " assert candidate(n = 25,queries = [[0, 10], [5, 20], [10, 22], [15, 23], [20, 24], [2, 12], [7, 17], [12, 21]]) == [15, 10, 4, 4, 4, 4, 4, 4]\n", " assert candidate(n = 20,queries = [[0, 19], [0, 18], [1, 17], [2, 16], [3, 15], [4, 14], [5, 13], [6, 12], [7, 11], [8, 10]]) == [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\n", " assert candidate(n = 20,queries = [[0, 15], [1, 16], [2, 17], [3, 18], [4, 19], [5, 13], [6, 14], [7, 12], [8, 11], [9, 10]]) == [5, 5, 5, 5, 5, 5, 5, 5, 5, 5]\n", " assert candidate(n = 25,queries = [[0, 15], [1, 16], [2, 17], [3, 18], [4, 19], [5, 20], [6, 21], [7, 22], [8, 23], [9, 24], [10, 18], [11, 19], [12, 20], [13, 21], [14, 22], [0, 8], [2, 16], [4, 18], [6, 20], [8, 22]]) == [10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 3, 3, 3, 3, 3]\n", " assert candidate(n = 50,queries = [[0, 20], [5, 30], [10, 36], [15, 40], [20, 44], [25, 47], [30, 48], [35, 49], [40, 46], [1, 21], [6, 31], [11, 37], [16, 41], [21, 43], [26, 38], [31, 39], [36, 45]]) == [30, 25, 24, 24, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7]\n", " assert candidate(n = 25,queries = [[0, 24], [0, 13], [1, 23], [2, 22], [3, 21], [4, 20], [5, 19], [6, 18], [7, 17], [8, 16], [9, 15], [10, 14], [11, 12]]) == [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\n", " assert candidate(n = 20,queries = [[0, 10], [1, 5], [2, 8], [3, 7], [4, 6], [5, 15], [6, 12], [7, 18], [8, 16], [9, 19]]) == [10, 10, 10, 10, 10, 7, 7, 6, 6, 5]\n", "\n" ] } ], "source": [ "print(problems[0][\"test\"])" ] }, { "cell_type": "code", "execution_count": 8, "id": "f807e5d4", "metadata": { "execution": { "iopub.execute_input": "2026-10-29T22:54:28.369063Z", "iopub.status.busy": "2026-10-29T22:54:28.368914Z", "iopub.status.idle": "2026-10-29T22:54:28.371432Z", "shell.execute_reply": "2026-10-29T22:54:28.371033Z" } }, "outputs": [], "source": [ "tasks = [\n", " {\"id\": p[\"task_id\"], \"prompt\": p[\"query\"],\n", " \"metadata\": {\"difficulty\": p[\"difficulty\"].lower()}}\n", " for p in problems\n", "]" ] }, { "cell_type": "markdown", "id": "25f8464a", "metadata": {}, "source": [ "## Start/resume answers generation run\n", "\n", "`TaskRunner` saves answers on the device and continues after the notebook\n", "disconnects. Starting the same named run processes unfinished tasks. The engine\n", "is shared across tasks, with a fresh conversation for each solution." ] }, { "cell_type": "code", "execution_count": 9, "id": "2ed657f7", "metadata": { "execution": { "iopub.execute_input": "2026-10-29T22:54:28.372874Z", "iopub.status.busy": "2026-10-29T22:54:28.372697Z", "iopub.status.idle": "2026-10-29T22:54:28.856004Z", "shell.execute_reply": "2026-10-29T22:54:28.854967Z" } }, "outputs": [], "source": [ "%there put tasks" ] }, { "cell_type": "code", "execution_count": 10, "id": "47fe83ff", "metadata": { "execution": { "iopub.execute_input": "2026-10-29T22:54:28.860761Z", "iopub.status.busy": "2026-10-29T22:54:28.860016Z", "iopub.status.idle": "2026-10-29T22:54:29.529298Z", "shell.execute_reply": "2026-10-29T22:54:29.528885Z" } }, "outputs": [], "source": [ "%%there\n", "from tasks_here import TaskRunner, TaskRunExistsError\n", "\n", "try:\n", " leetcode_runner = TaskRunner.create(RUN_NAME, max_attempts=3, metadata=settings)\n", "except TaskRunExistsError:\n", " leetcode_runner = TaskRunner.open(RUN_NAME)\n", "\n", "existing = [r[\"task\"] for r in leetcode_runner.results()]\n", "if not existing:\n", " leetcode_runner.add(tasks)\n" ] }, { "cell_type": "code", "execution_count": 11, "id": "03aa11c4", "metadata": { "execution": { "iopub.execute_input": "2026-10-29T22:54:29.531309Z", "iopub.status.busy": "2026-10-29T22:54:29.531064Z", "iopub.status.idle": "2026-10-29T22:54:29.654850Z", "shell.execute_reply": "2026-10-29T22:54:29.653991Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'name': 'leetcode-qwen-greedy-full-001', 'state': 'completed', 'total': 228, 'pending': 0, 'running': 0, 'completed': 228, 'failed': 0, 'progress': 1.0, 'current': None, 'pause_requested': False, 'max_attempts': 3, 'exclusive': True, 'created_at': '2026-09-17T21:48:35.563260+00:00', 'updated_at': '2026-10-29T22:54:29.704064+00:00', 'metadata': {}, 'worker_error': None, 'path': '/data/user/0/me.herethere.pythonhere_dev/files/task-runs/leetcode-qwen-greedy-full-001'}" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n" ] } ], "source": [ "%%there\n", "import litert_lm\n", "from ml_here import require_model\n", "\n", "def leetcode_setup():\n", " global leetcode_engine\n", " if globals().get(\"leetcode_engine\") is not None:\n", " raise RuntimeError(\"Close previous leetcode_engine using the last cell before resuming\")\n", " leetcode_engine = litert_lm.Engine(\n", " require_model(settings[\"model_repo\"], settings[\"model_file\"]),\n", " backend=litert_lm.Backend.GPU(), enable_benchmark=True,\n", " )\n", " return leetcode_engine\n", "\n", "def leetcode_execute(task, engine):\n", " with engine.create_conversation(\n", " sampler_config=litert_lm.SamplerConfig(**settings[\"sampler\"]),\n", " max_output_tokens=settings[\"max_output_tokens\"],\n", " ) as conversation:\n", " response = conversation.send_message(task[\"prompt\"])\n", " try:\n", " info = conversation.get_benchmark_info()\n", " benchmark = {k: getattr(info, k) for k in (\n", " \"init_time_in_second\", \"time_to_first_token_in_second\",\n", " \"last_prefill_token_count\", \"last_prefill_tokens_per_second\",\n", " \"last_decode_token_count\", \"last_decode_tokens_per_second\",\n", " )}\n", " benchmark_error = None\n", " except Exception as exc:\n", " benchmark, benchmark_error = None, str(exc)\n", " return {\n", " \"output\": \"\".join(b[\"text\"] for b in response[\"content\"] if \"text\" in b),\n", " \"benchmark\": benchmark, \"benchmark_error\": benchmark_error,\n", " }\n", "\n", "if not leetcode_runner.is_running():\n", " leetcode_runner.start(leetcode_execute, setup=leetcode_setup)\n", "print(leetcode_runner.status())" ] }, { "cell_type": "markdown", "id": "bc4cf090", "metadata": {}, "source": [ "## Check progress and fetch the results\n", "\n", "Completed answers can be fetched and scored while generation continues.\n", "After reconnecting, rerun the configuration and run-opening cells to retrieve\n", "saved answers without starting generation again." ] }, { "cell_type": "code", "execution_count": 12, "id": "f765ee3a", "metadata": { "execution": { "iopub.execute_input": "2026-10-29T22:54:29.659099Z", "iopub.status.busy": "2026-10-29T22:54:29.658726Z", "iopub.status.idle": "2026-10-29T22:54:29.744703Z", "shell.execute_reply": "2026-10-29T22:54:29.743467Z" } }, "outputs": [ { "data": { "text/plain": [ "{'name': 'leetcode-qwen-greedy-full-001',\n", " 'state': 'completed',\n", " 'total': 228,\n", " 'pending': 0,\n", " 'running': 0,\n", " 'completed': 228,\n", " 'failed': 0,\n", " 'progress': 1.0,\n", " 'current': None,\n", " 'pause_requested': False,\n", " 'max_attempts': 3,\n", " 'exclusive': True,\n", " 'created_at': '2026-09-17T21:48:35.563260+00:00',\n", " 'updated_at': '2026-10-29T22:54:29.704064+00:00',\n", " 'metadata': {},\n", " 'worker_error': None,\n", " 'path': '/data/user/0/me.herethere.pythonhere_dev/files/task-runs/leetcode-qwen-greedy-full-001'}" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "%there get leetcode_runner.status()" ] }, { "cell_type": "code", "execution_count": 13, "id": "af696694", "metadata": { "execution": { "iopub.execute_input": "2026-10-29T22:54:29.748969Z", "iopub.status.busy": "2026-10-29T22:54:29.748585Z", "iopub.status.idle": "2026-10-29T22:54:30.196306Z", "shell.execute_reply": "2026-10-29T22:54:30.195932Z" } }, "outputs": [], "source": [ "results = %there get leetcode_runner.results()" ] }, { "cell_type": "code", "execution_count": 14, "id": "de548995", "metadata": { "execution": { "iopub.execute_input": "2026-10-29T22:54:30.198011Z", "iopub.status.busy": "2026-10-29T22:54:30.197807Z", "iopub.status.idle": "2026-10-29T22:54:30.201048Z", "shell.execute_reply": "2026-10-29T22:54:30.200734Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'attempt_count': 1,\n", " 'attempts': [{'duration': 65.69324554200284,\n", " 'error': None,\n", " 'finished_at': '2026-09-18T17:05:00.478428+00:00',\n", " 'number': 1,\n", " 'started_at': '2026-09-18T17:03:54.781012+00:00'}],\n", " 'completed_at': '2026-09-18T17:05:00.478464+00:00',\n", " 'created_at': '2026-09-17T21:48:35.616171+00:00',\n", " 'duration': 65.69324554200284,\n", " 'error': None,\n", " 'failure_count': 0,\n", " 'id': 'minimum-operations-to-make-elements-within-k-subarrays-equal',\n", " 'result': {'benchmark': {'init_time_in_second': 35.336318156,\n", " 'last_decode_token_count': 126,\n", " 'last_decode_tokens_per_second': 3.1213213294337807,\n", " 'last_prefill_token_count': 610,\n", " 'last_prefill_tokens_per_second': 24.098436172011727,\n", " 'time_to_first_token_in_second': 25.63322298775},\n", " 'output': '```python\\n'\n", " 'class Solution:\\n'\n", " ' def minOperations(self, nums: List[int], x: int, k: '\n", " 'int) -> int:\\n'\n", " ' n = len(nums)\\n'\n", " ' count = 0\\n'\n", " ' operations = 0\\n'\n", " ' \\n'\n", " ' for i in range(n):\\n'\n", " ' if nums[i] == x:\\n'\n", " ' count += 1\\n'\n", " ' if count == k:\\n'\n", " ' operations += n - i - 1\\n'\n", " ' count = 0\\n'\n", " ' elif count > k:\\n'\n", " ' operations += n - i - 1\\n'\n", " ' count = 1\\n'\n", " ' else:\\n'\n", " ' count = 0\\n'\n", " ' \\n'\n", " ' return operations\\n'\n", " '```'},\n", " 'result_present': True,\n", " 'status': 'done',\n", " 'task': {'id': 'minimum-operations-to-make-elements-within-k-subarrays-equal',\n", " 'metadata': {'difficulty': 'Hard',\n", " 'task_id': 'minimum-operations-to-make-elements-within-k-subarrays-equal'},\n", " 'prompt': 'You are an expert Python programmer. You will be given a '\n", " 'question (problem specification) and will generate a '\n", " 'correct Python program that matches the specification and '\n", " 'passes all tests.\\n'\n", " '\\n'\n", " '### Question:\\n'\n", " 'You are given an integer array nums and two integers, x '\n", " 'and k. You can perform the following operation any number '\n", " 'of times (including zero):\\n'\n", " '\\n'\n", " 'Increase or decrease any element of nums by 1.\\n'\n", " '\\n'\n", " 'Return the minimum number of operations needed to have at '\n", " 'least k non-overlapping subarrays of size exactly x in '\n", " 'nums, where all elements within each subarray are equal.\\n'\n", " '\\xa0\\n'\n", " 'Example 1:\\n'\n", " '\\n'\n", " 'Input: nums = [5,-2,1,3,7,3,6,4,-1], x = 3, k = 2\\n'\n", " 'Output: 8\\n'\n", " 'Explanation:\\n'\n", " '\\n'\n", " 'Use 3 operations to add 3 to nums[1] and use 2 operations '\n", " 'to subtract 2 from nums[3]. The resulting array is [5, 1, '\n", " '1, 1, 7, 3, 6, 4, -1].\\n'\n", " 'Use 1 operation to add 1 to nums[5] and use 2 operations '\n", " 'to subtract 2 from nums[6]. The resulting array is [5, 1, '\n", " '1, 1, 7, 4, 4, 4, -1].\\n'\n", " 'Now, all elements within each subarray [1, 1, 1] (from '\n", " 'indices 1 to 3) and [4, 4, 4] (from indices 5 to 7) are '\n", " 'equal. Since 8 total operations were used, 8 is the '\n", " 'output.\\n'\n", " '\\n'\n", " '\\n'\n", " 'Example 2:\\n'\n", " '\\n'\n", " 'Input: nums = [9,-2,-2,-2,1,5], x = 2, k = 2\\n'\n", " 'Output: 3\\n'\n", " 'Explanation:\\n'\n", " '\\n'\n", " 'Use 3 operations to subtract 3 from nums[4]. The '\n", " 'resulting array is [9, -2, -2, -2, -2, 5].\\n'\n", " 'Now, all elements within each subarray [-2, -2] (from '\n", " 'indices 1 to 2) and [-2, -2] (from indices 3 to 4) are '\n", " 'equal. Since 3 operations were used, 3 is the output.\\n'\n", " '\\n'\n", " '\\n'\n", " '\\xa0\\n'\n", " 'Constraints:\\n'\n", " '\\n'\n", " '2 <= nums.length <= 105\\n'\n", " '-106 <= nums[i] <= 106\\n'\n", " '2 <= x <= nums.length\\n'\n", " '1 <= k <= 15\\n'\n", " '2 <= k * x <= nums.length\\n'\n", " '\\n'\n", " '\\n'\n", " '\\n'\n", " '### Format: You will use the following starter code to '\n", " 'write the solution to the problem and enclose your code '\n", " 'within delimiters.\\n'\n", " '```python\\n'\n", " 'class Solution:\\n'\n", " ' def minOperations(self, nums: List[int], x: int, k: '\n", " 'int) -> int:\\n'\n", " ' \\n'\n", " '```\\n'\n", " '\\n'\n", " '### Answer: (use the provided format with backticks)\\n'},\n", " 'total_duration': 65.69324554200284}\n" ] } ], "source": [ "pp(results[-1])" ] }, { "cell_type": "markdown", "id": "dfd89637", "metadata": {}, "source": [ "## Score fetched answers locally\n", "\n", "We build a Pydantic Evals `Dataset` from completed answers. The custom\n", "`LeetCodePass` evaluator extracts Python code and runs each solution with its\n", "tests in a local subprocess, with a 10-second timeout. The difficulty summary\n", "separates passes, assertion failures, invalid answers, syntax errors, runtime\n", "errors, and timeouts.\n", "\n", "`generation_s` is TaskRunner's elapsed time around the final `leetcode_execute`\n", "call on the device, including conversation setup and cleanup but excluding\n", "engine loading and earlier retries. `decode_tokens_s` is the decoding speed\n", "reported by LiteRT's benchmark." ] }, { "cell_type": "code", "execution_count": 15, "id": "dfbcd2ca", "metadata": { "execution": { "iopub.execute_input": "2026-10-29T22:54:30.202532Z", "iopub.status.busy": "2026-10-29T22:54:30.202389Z", "iopub.status.idle": "2026-10-29T22:54:31.879920Z", "shell.execute_reply": "2026-10-29T22:54:31.879440Z" } }, "outputs": [], "source": [ "import sys\n", "import json\n", "import re\n", "import subprocess\n", "from dataclasses import dataclass\n", "\n", "from pydantic_evals import Case, Dataset, increment_eval_metric\n", "from pydantic_evals.evaluators import Evaluator, EvaluationReason\n", "\n", "\n", "def extract_python_code(text):\n", " fences = re.findall(r\"```([^\\n`]*)\\n(.*?)```\", text, flags=re.DOTALL)\n", " if fences:\n", " if len(fences) != 1 or fences[0][0].strip().lower() not in {\"\", \"python\", \"py\"}:\n", " raise ValueError(\"Expected one Python code block\")\n", " code = fences[0][1].strip()\n", " elif \"```\" in text:\n", " raise ValueError(\"Incomplete code block\")\n", " else:\n", " code = text.strip()\n", " if not code:\n", " raise ValueError(\"Empty code\")\n", " return code\n", "\n", "\n", "def execute_problem(problem, code, timeout):\n", " program = \"\\n\".join([\n", " problem[\"prompt\"], code, problem[\"test\"],\n", " f\"check({problem['entry_point']})\",\n", " ])\n", " # Report success only after the complete program and tests return.\n", " harness = f'''\n", "import json\n", "try:\n", " exec(compile({program!r}, \"\", \"exec\"), {{}})\n", "except SyntaxError as exc:\n", " result = {{\"outcome\": \"syntax_error\", \"error\": str(exc)}}\n", "except AssertionError as exc:\n", " result = {{\"outcome\": \"test_failed\", \"error\": \"AssertionError: \" + str(exc)}}\n", "except BaseException as exc:\n", " result = {{\"outcome\": \"runtime_error\", \"error\": type(exc).__name__ + \": \" + str(exc)}}\n", "else:\n", " result = {{\"outcome\": \"passed\"}}\n", "print(\"__TEST_RESULT__\" + json.dumps(result))\n", "'''\n", " try:\n", " process = subprocess.run(\n", " [sys.executable, \"-I\", \"-\"], input=harness,\n", " text=True, capture_output=True, timeout=timeout,\n", " )\n", " except subprocess.TimeoutExpired:\n", " return {\"outcome\": \"timeout\", \"error\": f\"Exceeded {timeout:g} seconds\"}\n", " for line in reversed(process.stdout.splitlines()):\n", " if line.startswith(\"__TEST_RESULT__\"):\n", " return json.loads(line[len(\"__TEST_RESULT__\"):])\n", " return {\"outcome\": \"runtime_error\", \"error\": process.stderr.strip() or \"Tests did not finish\"}\n", "\n", "\n", "@dataclass\n", "class LeetCodePass(Evaluator):\n", " problems: dict\n", " timeout: float = 10.0\n", "\n", " def evaluate(self, ctx):\n", " try:\n", " code = extract_python_code(ctx.output)\n", " except ValueError as exc:\n", " return {\n", " \"passed\": EvaluationReason(False, str(exc)),\n", " \"failure\": \"invalid\",\n", " }\n", "\n", " problem = self.problems[ctx.inputs]\n", " result = execute_problem(problem, code, self.timeout)\n", " if result[\"outcome\"] == \"passed\":\n", " return {\"passed\": True}\n", " return {\n", " \"passed\": EvaluationReason(False, result.get(\"error\") or result[\"outcome\"]),\n", " \"failure\": result[\"outcome\"],\n", " }" ] }, { "cell_type": "code", "execution_count": 16, "id": "76b07be4", "metadata": { "execution": { "iopub.execute_input": "2026-10-29T22:54:31.881878Z", "iopub.status.busy": "2026-10-29T22:54:31.881578Z", "iopub.status.idle": "2026-10-29T22:54:31.885688Z", "shell.execute_reply": "2026-10-29T22:54:31.885400Z" } }, "outputs": [], "source": [ "from collections import Counter\n", "from pydantic_evals.evaluators import ReportEvaluator, ReportEvaluatorContext\n", "from pydantic_evals.reporting.analyses import TableResult\n", "\n", "\n", "@dataclass\n", "class LeetCodeSummary(ReportEvaluator):\n", " def evaluate(self, ctx: ReportEvaluatorContext):\n", " categories = [\"passed\", \"test_failed\", \"invalid\", \"syntax_error\",\n", " \"runtime_error\", \"timeout\"]\n", " rows = []\n", " for difficulty in [\"all\", \"easy\", \"medium\", \"hard\"]:\n", " cases = [c for c in ctx.report.cases\n", " if difficulty == \"all\" or c.metadata[\"difficulty\"] == difficulty]\n", " if not cases:\n", " continue\n", " counts = Counter()\n", " for case in cases:\n", " if case.assertions[\"passed\"].value:\n", " counts[\"passed\"] += 1\n", " else:\n", " counts[case.labels[\"failure\"].value] += 1\n", " rows.append([difficulty, len(cases),\n", " *[round(100 * counts[name] / len(cases), 1)\n", " for name in categories]])\n", " return TableResult(\n", " title=\"LeetCode results by difficulty\",\n", " columns=[\"Difficulty\", \"N\", \"Passed %\", \"Assertion %\", \"Invalid %\",\n", " \"Syntax %\", \"Runtime %\", \"Timeout %\"],\n", " rows=rows,\n", " )" ] }, { "cell_type": "code", "execution_count": 17, "id": "54829875", "metadata": { "execution": { "iopub.execute_input": "2026-10-29T22:54:31.887247Z", "iopub.status.busy": "2026-10-29T22:54:31.887028Z", "iopub.status.idle": "2026-10-29T22:56:23.761963Z", "shell.execute_reply": "2026-10-29T22:56:23.761509Z" } }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "9a90c011045f438cb2fb9e8e7bf6a1e7", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Output()" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
                                         Evaluation Summary: saved_answer                                          \n",
       "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃ Case ID                        ┃ Labels                 ┃ Metrics               ┃ Assertions                    ┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│ shortest-distance-after-road-… │ failure: runtime_error │ generation_s: 120.0   │ passed: ✗                     │\n",
       "│                                │                        │ decode_tokens_s: 3.08 │   Reason: ValueError:         │\n",
       "│                                │                        │                       │ list.remove(x): x not in list │\n",
       "│                                │                        │                       │                               │\n",
       "│                                │                        │                       │                               │\n",
       "├────────────────────────────────┼────────────────────────┼───────────────────────┼───────────────────────────────┤\n",
       "│ shortest-distance-after-road-… │ failure: test_failed   │ generation_s: 229.4   │ passed: ✗                     │\n",
       "│                                │                        │ decode_tokens_s: 3.15 │   Reason: AssertionError:     │\n",
       "│                                │                        │                       │                               │\n",
       "│                                │                        │                       │                               │\n",
       "├────────────────────────────────┼────────────────────────┼───────────────────────┼───────────────────────────────┤\n",
       "│ number-of-subsequences-with-o… │ failure: test_failed   │ generation_s: 63.0    │ passed: ✗                     │\n",
       "│                                │                        │ decode_tokens_s: 3.13 │   Reason: AssertionError:     │\n",
       "│                                │                        │                       │                               │\n",
       "│                                │                        │                       │                               │\n",
       "├────────────────────────────────┼────────────────────────┼───────────────────────┼───────────────────────────────┤\n",
       "│ snake-in-matrix                │ failure: test_failed   │ generation_s: 86.2    │ passed: ✗                     │\n",
       "│                                │                        │ decode_tokens_s: 3.15 │   Reason: AssertionError:     │\n",
       "│                                │                        │                       │                               │\n",
       "│                                │                        │                       │                               │\n",
       "├────────────────────────────────┼────────────────────────┼───────────────────────┼───────────────────────────────┤\n",
       "│ count-the-number-of-good-nodes │ failure: test_failed   │ generation_s: 71.9    │ passed: ✗                     │\n",
       "│                                │                        │ decode_tokens_s: 3.16 │   Reason: AssertionError:     │\n",
       "│                                │                        │                       │                               │\n",
       "│                                │                        │                       │                               │\n",
       "├────────────────────────────────┼────────────────────────┼───────────────────────┼───────────────────────────────┤\n",
       "│ find-the-count-of-monotonic-p… │ failure: runtime_error │ generation_s: 72.0    │ passed: ✗                     │\n",
       "│                                │                        │ decode_tokens_s: 3.17 │   Reason: AttributeError:     │\n",
       "│                                │                        │                       │ 'builtin_function_or_method'  │\n",
       "│                                │                        │                       │ object has no attribute       │\n",
       "│                                │                        │                       │ 'bisect_left'                 │\n",
       "│                                │                        │                       │                               │\n",
       "│                                │                        │                       │                               │\n",
       "├────────────────────────────────┼────────────────────────┼───────────────────────┼───────────────────────────────┤\n",
       "│ find-the-count-of-monotonic-p… │ failure: test_failed   │ generation_s: 95.1    │ passed: ✗                     │\n",
       "│                                │                        │ decode_tokens_s: 3.15 │   Reason: AssertionError:     │\n",
       "│                                │                        │                       │                               │\n",
       "│                                │                        │                       │                               │\n",
       "├────────────────────────────────┼────────────────────────┼───────────────────────┼───────────────────────────────┤\n",
       "│ construct-string-with-minimum… │ failure: runtime_error │ generation_s: 68.2    │ passed: ✗                     │\n",
       "│                                │                        │ decode_tokens_s: 3.15 │   Reason: IndexError: list    │\n",
       "│                                │                        │                       │ index out of range            │\n",
       "│                                │                        │                       │                               │\n",
       "│                                │                        │                       │                               │\n",
       "└────────────────────────────────┴────────────────────────┴───────────────────────┴───────────────────────────────┘\n",
       "
\n"
      ],
      "text/plain": [
       "\u001b[3m                                         Evaluation Summary: saved_answer                                          \u001b[0m\n",
       "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1mCase ID                       \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mLabels                \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mMetrics              \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mAssertions                   \u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│\u001b[1m \u001b[0m\u001b[1mshortest-distance-after-road-…\u001b[0m\u001b[1m \u001b[0m│ failure: runtime_error │ generation_s: 120.0   │ passed: \u001b[31m✗\u001b[0m                     │\n",
       "│\u001b[1m                                \u001b[0m│                        │ decode_tokens_s: 3.08 │   Reason: ValueError:         │\n",
       "│\u001b[1m                                \u001b[0m│                        │                       │ list.remove(x): x not in list │\n",
       "│\u001b[1m                                \u001b[0m│                        │                       │                               │\n",
       "│\u001b[1m                                \u001b[0m│                        │                       │                               │\n",
       "├────────────────────────────────┼────────────────────────┼───────────────────────┼───────────────────────────────┤\n",
       "│\u001b[1m \u001b[0m\u001b[1mshortest-distance-after-road-…\u001b[0m\u001b[1m \u001b[0m│ failure: test_failed   │ generation_s: 229.4   │ passed: \u001b[31m✗\u001b[0m                     │\n",
       "│\u001b[1m                                \u001b[0m│                        │ decode_tokens_s: 3.15 │   Reason: AssertionError:     │\n",
       "│\u001b[1m                                \u001b[0m│                        │                       │                               │\n",
       "│\u001b[1m                                \u001b[0m│                        │                       │                               │\n",
       "├────────────────────────────────┼────────────────────────┼───────────────────────┼───────────────────────────────┤\n",
       "│\u001b[1m \u001b[0m\u001b[1mnumber-of-subsequences-with-o…\u001b[0m\u001b[1m \u001b[0m│ failure: test_failed   │ generation_s: 63.0    │ passed: \u001b[31m✗\u001b[0m                     │\n",
       "│\u001b[1m                                \u001b[0m│                        │ decode_tokens_s: 3.13 │   Reason: AssertionError:     │\n",
       "│\u001b[1m                                \u001b[0m│                        │                       │                               │\n",
       "│\u001b[1m                                \u001b[0m│                        │                       │                               │\n",
       "├────────────────────────────────┼────────────────────────┼───────────────────────┼───────────────────────────────┤\n",
       "│\u001b[1m \u001b[0m\u001b[1msnake-in-matrix               \u001b[0m\u001b[1m \u001b[0m│ failure: test_failed   │ generation_s: 86.2    │ passed: \u001b[31m✗\u001b[0m                     │\n",
       "│\u001b[1m                                \u001b[0m│                        │ decode_tokens_s: 3.15 │   Reason: AssertionError:     │\n",
       "│\u001b[1m                                \u001b[0m│                        │                       │                               │\n",
       "│\u001b[1m                                \u001b[0m│                        │                       │                               │\n",
       "├────────────────────────────────┼────────────────────────┼───────────────────────┼───────────────────────────────┤\n",
       "│\u001b[1m \u001b[0m\u001b[1mcount-the-number-of-good-nodes\u001b[0m\u001b[1m \u001b[0m│ failure: test_failed   │ generation_s: 71.9    │ passed: \u001b[31m✗\u001b[0m                     │\n",
       "│\u001b[1m                                \u001b[0m│                        │ decode_tokens_s: 3.16 │   Reason: AssertionError:     │\n",
       "│\u001b[1m                                \u001b[0m│                        │                       │                               │\n",
       "│\u001b[1m                                \u001b[0m│                        │                       │                               │\n",
       "├────────────────────────────────┼────────────────────────┼───────────────────────┼───────────────────────────────┤\n",
       "│\u001b[1m \u001b[0m\u001b[1mfind-the-count-of-monotonic-p…\u001b[0m\u001b[1m \u001b[0m│ failure: runtime_error │ generation_s: 72.0    │ passed: \u001b[31m✗\u001b[0m                     │\n",
       "│\u001b[1m                                \u001b[0m│                        │ decode_tokens_s: 3.17 │   Reason: AttributeError:     │\n",
       "│\u001b[1m                                \u001b[0m│                        │                       │ 'builtin_function_or_method'  │\n",
       "│\u001b[1m                                \u001b[0m│                        │                       │ object has no attribute       │\n",
       "│\u001b[1m                                \u001b[0m│                        │                       │ 'bisect_left'                 │\n",
       "│\u001b[1m                                \u001b[0m│                        │                       │                               │\n",
       "│\u001b[1m                                \u001b[0m│                        │                       │                               │\n",
       "├────────────────────────────────┼────────────────────────┼───────────────────────┼───────────────────────────────┤\n",
       "│\u001b[1m \u001b[0m\u001b[1mfind-the-count-of-monotonic-p…\u001b[0m\u001b[1m \u001b[0m│ failure: test_failed   │ generation_s: 95.1    │ passed: \u001b[31m✗\u001b[0m                     │\n",
       "│\u001b[1m                                \u001b[0m│                        │ decode_tokens_s: 3.15 │   Reason: AssertionError:     │\n",
       "│\u001b[1m                                \u001b[0m│                        │                       │                               │\n",
       "│\u001b[1m                                \u001b[0m│                        │                       │                               │\n",
       "├────────────────────────────────┼────────────────────────┼───────────────────────┼───────────────────────────────┤\n",
       "│\u001b[1m \u001b[0m\u001b[1mconstruct-string-with-minimum…\u001b[0m\u001b[1m \u001b[0m│ failure: runtime_error │ generation_s: 68.2    │ passed: \u001b[31m✗\u001b[0m                     │\n",
       "│\u001b[1m                                \u001b[0m│                        │ decode_tokens_s: 3.15 │   Reason: IndexError: list    │\n",
       "│\u001b[1m                                \u001b[0m│                        │                       │ index out of range            │\n",
       "│\u001b[1m                                \u001b[0m│                        │                       │                               │\n",
       "│\u001b[1m                                \u001b[0m│                        │                       │                               │\n",
       "└────────────────────────────────┴────────────────────────┴───────────────────────┴───────────────────────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "
                               LeetCode results by difficulty                               \n",
       "┏━━━━━━━━━━━━┳━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┓\n",
       "┃ Difficulty ┃ N   ┃ Passed % ┃ Assertion % ┃ Invalid % ┃ Syntax % ┃ Runtime % ┃ Timeout % ┃\n",
       "┡━━━━━━━━━━━━╇━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━┩\n",
       "│ all        │ 228 │ 10.1     │ 61.8        │ 6.1       │ 5.3      │ 15.4      │ 1.3       │\n",
       "├────────────┼─────┼──────────┼─────────────┼───────────┼──────────┼───────────┼───────────┤\n",
       "│ easy       │ 48  │ 31.2     │ 54.2        │ 6.2       │ 4.2      │ 4.2       │ 0.0       │\n",
       "├────────────┼─────┼──────────┼─────────────┼───────────┼──────────┼───────────┼───────────┤\n",
       "│ medium     │ 101 │ 5.9      │ 71.3        │ 6.9       │ 3.0      │ 11.9      │ 1.0       │\n",
       "├────────────┼─────┼──────────┼─────────────┼───────────┼──────────┼───────────┼───────────┤\n",
       "│ hard       │ 79  │ 2.5      │ 54.4        │ 5.1       │ 8.9      │ 26.6      │ 2.5       │\n",
       "└────────────┴─────┴──────────┴─────────────┴───────────┴──────────┴───────────┴───────────┘\n",
       "
\n"
      ],
      "text/plain": [
       "\u001b[3m                               LeetCode results by difficulty                               \u001b[0m\n",
       "┏━━━━━━━━━━━━┳━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1mDifficulty\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mN  \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mPassed %\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mAssertion %\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mInvalid %\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mSyntax %\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mRuntime %\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mTimeout %\u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━━━╇━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━┩\n",
       "│ all        │ 228 │ 10.1     │ 61.8        │ 6.1       │ 5.3      │ 15.4      │ 1.3       │\n",
       "├────────────┼─────┼──────────┼─────────────┼───────────┼──────────┼───────────┼───────────┤\n",
       "│ easy       │ 48  │ 31.2     │ 54.2        │ 6.2       │ 4.2      │ 4.2       │ 0.0       │\n",
       "├────────────┼─────┼──────────┼─────────────┼───────────┼──────────┼───────────┼───────────┤\n",
       "│ medium     │ 101 │ 5.9      │ 71.3        │ 6.9       │ 3.0      │ 11.9      │ 1.0       │\n",
       "├────────────┼─────┼──────────┼─────────────┼───────────┼──────────┼───────────┼───────────┤\n",
       "│ hard       │ 79  │ 2.5      │ 54.4        │ 5.1       │ 8.9      │ 26.6      │ 2.5       │\n",
       "└────────────┴─────┴──────────┴─────────────┴───────────┴──────────┴───────────┴───────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "problems_by_id = {p[\"task_id\"]: p for p in problems}\n",
    "completed = {r[\"id\"]: r for r in results if r[\"status\"] == \"done\"}\n",
    "for case_id, record in completed.items():\n",
    "    if case_id not in problems_by_id or record[\"task\"][\"prompt\"] != problems_by_id[case_id][\"query\"]:\n",
    "        raise ValueError(\"Fetched tasks differ from selected dataset\")\n",
    "\n",
    "def saved_answer(case_id):\n",
    "    record = completed[case_id]\n",
    "    if record.get(\"duration\") is not None:\n",
    "        increment_eval_metric(\"generation_s\", record[\"duration\"])\n",
    "    benchmark = record[\"result\"].get(\"benchmark\") or {}\n",
    "    speed = benchmark.get(\"last_decode_tokens_per_second\")\n",
    "    if speed is not None:\n",
    "        increment_eval_metric(\"decode_tokens_s\", speed)\n",
    "    return record[\"result\"][\"output\"]\n",
    "\n",
    "dataset = Dataset(\n",
    "    name=\"pythonhere-leetcode\",\n",
    "    cases=[Case(name=p[\"task_id\"], inputs=p[\"task_id\"],\n",
    "                metadata={\"difficulty\": p[\"difficulty\"].lower()})\n",
    "           for p in problems if p[\"task_id\"] in completed],\n",
    "    evaluators=[LeetCodePass(problems_by_id)],\n",
    "    report_evaluators=[LeetCodeSummary()],\n",
    ")\n",
    "if not dataset.cases:\n",
    "    raise RuntimeError(\"No completed answers; check device status/errors\")\n",
    "report = await dataset.evaluate(saved_answer, max_concurrency=1)\n",
    "\n",
    "# report.print(include_durations=False, include_averages=True, include_reasons=True)\n",
    "from dataclasses import replace\n",
    "# Show the first eight cases; LeetCodeSummary covers all scored cases.\n",
    "preview = replace(report, cases=report.cases[:8])\n",
    "preview.print(include_durations=False, include_averages=False, include_analyses=True, include_reasons=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "560485a1",
   "metadata": {},
   "source": [
    "Show bad answers that are missing `class Solution`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "0d70e9f2",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-29T22:56:25.325908Z",
     "iopub.status.busy": "2026-10-29T22:56:25.325570Z",
     "iopub.status.idle": "2026-10-29T22:56:25.336102Z",
     "shell.execute_reply": "2026-10-29T22:56:25.335674Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "
                                              Unexpected outputs (12)                                              \n",
       "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃                                                      # ┃ Output                                                 ┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│             find-the-largest-palindrome-divisible-by-k │ '```python\\n99999\\n](https://leetcode.com/problems/lar │\n",
       "│                                                        │ gest-palindromic-number/discuss/100000/JavaC%2B%2BPyth │\n",
       "│                                                        │ on-3-99999)'                                           │\n",
       "│                 maximum-total-area-occupied-by-pistons │ 'duplicate'                                            │\n",
       "│                      find-the-original-typed-string-ii │ 'duplicate =7\\n Sponsored by Alibaba Cloud.'           │\n",
       "│                count-connected-components-in-lcm-graph │ 'duplicate'                                            │\n",
       "│          maximize-amount-after-two-days-of-conversions │ 'To solve this problem, we need to determine the       │\n",
       "│                                                        │ maximum amount of currency we can obtain starting with │\n",
       "│                                                        │ a given initial currency on both days using the        │\n",
       "│                                                        │ provided conversion rates. The key is to use dynamic   │\n",
       "│                                                        │ programming to keep track of the maximum amount of     │\n",
       "│                                                        │ each currency we can obtain at each step.'             │\n",
       "│         smallest-substring-with-identical-characters-i │ 'inguire'                                              │\n",
       "│                             substring-matching-pattern │ 'inguence\\n Sponsored by Alibaba Cloud.'               │\n",
       "│ maximize-subarray-sum-after-removing-all-occurrences-… │ 'inguence\\npired.'                                     │\n",
       "│                            shortest-matching-substring │ 'duplicate'                                            │\n",
       "│       maximize-the-distance-between-points-on-a-square │ 'inguence\\n Sponsored by Alibaba Cloud.'               │\n",
       "│            minimum-cost-to-divide-array-into-subarrays │ 'inguire\\n Sponsored by Alibaba Cloud.'                │\n",
       "│                  maximize-active-section-with-trade-ii │ 'inguire\\n Sponsored by Alibaba Cloud.'                │\n",
       "└────────────────────────────────────────────────────────┴────────────────────────────────────────────────────────┘\n",
       "
\n"
      ],
      "text/plain": [
       "\u001b[3m                                              Unexpected outputs (12)                                              \u001b[0m\n",
       "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1m                                                     #\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mOutput                                                \u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│\u001b[2m \u001b[0m\u001b[2m            find-the-largest-palindrome-divisible-by-k\u001b[0m\u001b[2m \u001b[0m│ '```python\\n99999\\n](https://leetcode.com/problems/lar │\n",
       "│\u001b[2m                                                        \u001b[0m│ gest-palindromic-number/discuss/100000/JavaC%2B%2BPyth │\n",
       "│\u001b[2m                                                        \u001b[0m│ on-3-99999)'                                           │\n",
       "│\u001b[2m \u001b[0m\u001b[2m                maximum-total-area-occupied-by-pistons\u001b[0m\u001b[2m \u001b[0m│ 'duplicate'                                            │\n",
       "│\u001b[2m \u001b[0m\u001b[2m                     find-the-original-typed-string-ii\u001b[0m\u001b[2m \u001b[0m│ 'duplicate =7\\n Sponsored by Alibaba Cloud.'           │\n",
       "│\u001b[2m \u001b[0m\u001b[2m               count-connected-components-in-lcm-graph\u001b[0m\u001b[2m \u001b[0m│ 'duplicate'                                            │\n",
       "│\u001b[2m \u001b[0m\u001b[2m         maximize-amount-after-two-days-of-conversions\u001b[0m\u001b[2m \u001b[0m│ 'To solve this problem, we need to determine the       │\n",
       "│\u001b[2m                                                        \u001b[0m│ maximum amount of currency we can obtain starting with │\n",
       "│\u001b[2m                                                        \u001b[0m│ a given initial currency on both days using the        │\n",
       "│\u001b[2m                                                        \u001b[0m│ provided conversion rates. The key is to use dynamic   │\n",
       "│\u001b[2m                                                        \u001b[0m│ programming to keep track of the maximum amount of     │\n",
       "│\u001b[2m                                                        \u001b[0m│ each currency we can obtain at each step.'             │\n",
       "│\u001b[2m \u001b[0m\u001b[2m        smallest-substring-with-identical-characters-i\u001b[0m\u001b[2m \u001b[0m│ 'inguire'                                              │\n",
       "│\u001b[2m \u001b[0m\u001b[2m                            substring-matching-pattern\u001b[0m\u001b[2m \u001b[0m│ 'inguence\\n Sponsored by Alibaba Cloud.'               │\n",
       "│\u001b[2m \u001b[0m\u001b[2mmaximize-subarray-sum-after-removing-all-occurrences-…\u001b[0m\u001b[2m \u001b[0m│ 'inguence\\npired.'                                     │\n",
       "│\u001b[2m \u001b[0m\u001b[2m                           shortest-matching-substring\u001b[0m\u001b[2m \u001b[0m│ 'duplicate'                                            │\n",
       "│\u001b[2m \u001b[0m\u001b[2m      maximize-the-distance-between-points-on-a-square\u001b[0m\u001b[2m \u001b[0m│ 'inguence\\n Sponsored by Alibaba Cloud.'               │\n",
       "│\u001b[2m \u001b[0m\u001b[2m           minimum-cost-to-divide-array-into-subarrays\u001b[0m\u001b[2m \u001b[0m│ 'inguire\\n Sponsored by Alibaba Cloud.'                │\n",
       "│\u001b[2m \u001b[0m\u001b[2m                 maximize-active-section-with-trade-ii\u001b[0m\u001b[2m \u001b[0m│ 'inguire\\n Sponsored by Alibaba Cloud.'                │\n",
       "└────────────────────────────────────────────────────────┴────────────────────────────────────────────────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from rich.console import Console\n",
    "from rich.table import Table\n",
    "from rich.text import Text\n",
    "\n",
    "weird = [\n",
    "    r[\"result\"][\"output\"]\n",
    "    for r in results\n",
    "    if r[\"status\"] == \"done\"\n",
    "    and \"class Solution\" not in r[\"result\"][\"output\"]\n",
    "]\n",
    "\n",
    "table = Table(title=f\"Unexpected outputs ({len(weird)})\")\n",
    "table.add_column(\"#\", style=\"dim\", justify=\"right\")\n",
    "table.add_column(\"Output\", overflow=\"fold\")\n",
    "\n",
    "for r in results:\n",
    "    if r[\"status\"] == \"done\":\n",
    "        s = r[\"result\"][\"output\"]\n",
    "        if \"class Solution\" not in s:\n",
    "            table.add_row(r[\"id\"], Text(repr(s[:300])))\n",
    "\n",
    "Console().print(table)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5cb3f83b",
   "metadata": {},
   "source": [
    "Another bad answer from the video starts with `class Solution`, but keeps\n",
    "adding nested loops until the output is cut off. Show that problem's saved answer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "e7e5dfba",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-29T22:56:25.338118Z",
     "iopub.status.busy": "2026-10-29T22:56:25.337932Z",
     "iopub.status.idle": "2026-10-29T22:56:25.340953Z",
     "shell.execute_reply": "2026-10-29T22:56:25.340534Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "```python\n",
      "class Solution:\n",
      "    def answerString(self, word: str, numFriends: int) -> str:\n",
      "        n = len(word)\n",
      "        if n == 0:\n",
      "            return \"\"\n",
      "        \n",
      "        # Initialize the result with the entire word\n",
      "        result = word\n",
      "        \n",
      "        # Iterate over all possible splits\n",
      "        for i in range(1, n // numFriends + 1):\n",
      "            # Generate all possible splits of the word into numFriends parts\n",
      "            for j in range(i, n - i + 1, i):\n",
      "                # Generate all possible splits of the word into numFriends parts\n",
      "                for k in range(j, n - j + 1, i):\n",
      "                    # Generate all possible splits of the word into numFriends parts\n",
      "                    for l in range(k, n - k + 1, i):\n",
      "                        # Generate all possible splits of the word into numFriends parts\n",
      "                        for m in range(l, n - l + 1, i):\n",
      "                            # Generate all possible splits of the word into numFriends parts\n",
      "                            for n in range(m, n - m + 1, i):\n",
      "                                # Generate all possible splits of the word into numFriends parts\n",
      "                                for o in range(n, n - n + 1, i):\n",
      "                                    # Generate all possible splits of the word into numFriends parts\n",
      "                                    for p in range(o, n - o + 1, i):\n",
      "                                        # Generate all possible splits of the word into numFriends parts\n",
      "                                        for q in range(p, n - p + 1, i):\n",
      "                                            # Generate all possible splits of the word into numFriends parts\n",
      "                                            for r in range(q, n - q + 1, i):\n",
      "                                                # Generate all possible splits of the word into numFriends parts\n",
      "                                                for s in range(r, n - r + 1, i):\n",
      "                                                    # Generate all possible splits of the word into numFriends parts\n",
      "                                                    for t in range(s, n - s + 1, i):\n",
      "                                                        # Generate all possible splits of the word into numFriends parts\n",
      "                                                        for u in range(t, n - t + 1, i):\n",
      "                                                            # Generate all possible splits of the word into numFriends parts\n",
      "                                                            for v in range(u, n - u + 1, i):\n",
      "                                                                # Generate all possible splits of the word into numFriends parts\n",
      "                                                                for w in range(v, n - v + 1, i):\n",
      "                                                                    # Generate all possible splits of the word into numFriends parts\n",
      "                                                                    for x in range(w, n - w + 1, i):\n",
      "                                                                        # Generate all possible splits of the word into numFriends parts\n",
      "                                                                        for y in range(x, n - x + 1, i):\n",
      "                                                                            # Generate all possible splits of the word into numFriends parts\n",
      "                                                                            for z in range(y, n - y + 1, i):\n",
      "                                                                                # Generate all possible splits of the word into numFriends parts\n",
      "                                                                                for a in range(z, n - z + 1, i):\n",
      "                                                                                    # Generate all possible splits of the word into numFriends parts\n",
      "                                                                                    for b in range(a, n - a + 1, i):\n",
      "                                                                                        # Generate all possible splits of the word into numFriends parts\n",
      "                                                                                        for c in range(b, n - b + 1, i):\n",
      "                                                                                            # Generate all possible splits of the word into numFriends parts\n",
      "                                                                                            for d in range(c, n - c + 1, i):\n",
      "                                                                                                # Generate all possible splits of the word into numFriends parts\n",
      "                                                                                                for e in range(d, n - d + 1, i):\n",
      "                                                                                                    # Generate all possible splits of the word into numFriends parts\n",
      "                                                                                                    for f in range(e, n - e + 1, i):\n",
      "                                                                                                        # Generate all possible splits of the word into numFriends parts\n",
      "                                                                                                        for g in range(f, n - f + 1, i):\n",
      "                                                                                                            # Generate all possible splits of the word into numFriends parts\n",
      "                                                                                                            for h in range(g, n - g + 1, i):\n",
      "                                                                                                                # Generate all possible splits of the word into numFriends parts\n",
      "                                                                                                                for i in range(h, n - h + 1, i):\n",
      "                                                                                                                    # Generate all possible splits of the word into numFriends parts\n",
      "                                                                                                                    for j in range(i, n - i + 1, i):\n",
      "                                                                                                                        # Generate all possible splits of the word into numFriends parts\n",
      "                                                                                                                        for k in range(j, n - j + 1, i):\n",
      "                                                                                                                            # Generate all possible splits of the word into numFriends parts\n",
      "                                                                                                                            for l in range(k, n - k + 1, i):\n",
      "                                                                                                                                # Generate all possible splits of the word into numFriends parts\n",
      "                                                                                                                                for m in range(l, n - l + 1, i):\n",
      "                                                                                                                                    # Generate\n"
     ]
    }
   ],
   "source": [
    "example_id = \"find-the-lexicographically-largest-string-from-the-box-i\"\n",
    "if example_id in completed:\n",
    "    print(completed[example_id][\"result\"][\"output\"])\n",
    "else:\n",
    "    print(f\"No saved answer for {example_id} in this run.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a75db42",
   "metadata": {},
   "source": [
    "## Close the engine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "213706bd",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-29T22:56:25.342748Z",
     "iopub.status.busy": "2026-10-29T22:56:25.342571Z",
     "iopub.status.idle": "2026-10-29T22:56:25.536821Z",
     "shell.execute_reply": "2026-10-29T22:56:25.535529Z"
    }
   },
   "outputs": [],
   "source": [
    "%%there --worker\n",
    "if globals().get(\"leetcode_engine\") is not None:\n",
    "    leetcode_engine.close()\n",
    "    leetcode_engine = None"
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "default_lexer": "ipython3"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "there-ipython3",
   "version": "3.12.3"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {
     "0bc40d9cca8442558d30f134bc867177": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "2.0.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "2.0.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "2.0.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border_bottom": null,
       "border_left": null,
       "border_right": null,
       "border_top": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "9a90c011045f438cb2fb9e8e7bf6a1e7": {
      "model_module": "@jupyter-widgets/output",
      "model_module_version": "1.0.0",
      "model_name": "OutputModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/output",
       "_model_module_version": "1.0.0",
       "_model_name": "OutputModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/output",
       "_view_module_version": "1.0.0",
       "_view_name": "OutputView",
       "layout": "IPY_MODEL_0bc40d9cca8442558d30f134bc867177",
       "msg_id": "",
       "outputs": [
        {
         "data": {
          "text/html": "
Evaluating saved_answer ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 100% 0:00:02\n
\n",
          "text/plain": "Evaluating saved_answer \u001b[38;2;249;38;114m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[38;2;249;38;114m╸\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m0:00:02\u001b[0m\n"
         },
         "metadata": {},
         "output_type": "display_data"
        }
       ],
       "tabbable": null,
       "tooltip": null
      }
     }
    },
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}