{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "2863bd3a",
   "metadata": {},
   "source": [
    "# On-device AI with LiteRT\n",
    "\n",
    "PythonHere bundles the\n",
    "[LiteRT-LM](https://github.com/google-ai-edge/LiteRT-LM) Python library in its\n",
    "Android app, so `.litertlm` models can run directly on the device. After a\n",
    "model has been downloaded, inference does not require a network connection.\n",
    "\n",
    "PythonHere also provides\n",
    "[model helpers](https://github.com/b3b/pythonhere/tree/master/pythonhere/ml_here)\n",
    "for managing model storage and downloading model files from Hugging Face.\n",
    "This example covers every public helper, then loads a small language model and\n",
    "generates text with LiteRT-LM. Refer to the LiteRT-LM documentation for its\n",
    "full inference API, including sampling, streaming, and benchmarking.\n",
    "\n",
    "For an interactive vision-language example using the Android camera, see the\n",
    "[LiteRT multimodal lab](litert_multimodal_lab.md), the companion notebook for\n",
    "the video *Run multimodal AI on Android with Python and LiteRT*.\n",
    "\n",
    "## Connect to PythonHere\n",
    "\n",
    "Load the Jupyter extension and connect to a running PythonHere app:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "188792ab",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-09-06T10:01:30.865236Z",
     "iopub.status.busy": "2026-09-06T10:01:30.864833Z",
     "iopub.status.idle": "2026-09-06T10:01:31.622381Z",
     "shell.execute_reply": "2026-09-06T10:01:31.621129Z"
    }
   },
   "outputs": [],
   "source": [
    "%load_ext pythonhere\n",
    "%connect-there"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "304db7b2",
   "metadata": {},
   "source": [
    "## Inspect model storage\n",
    "\n",
    "`models_directory()` returns PythonHere's platform-specific model directory.\n",
    "`model_path()` builds the managed path for a particular Hugging Face model,\n",
    "and `discover_models()` lists models already present on the device."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e1614c0c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-09-06T10:01:31.627781Z",
     "iopub.status.busy": "2026-09-06T10:01:31.627287Z",
     "iopub.status.idle": "2026-09-06T10:01:32.254285Z",
     "shell.execute_reply": "2026-09-06T10:01:32.253809Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model directory: /storage/emulated/0/Android/data/me.herethere.pythonhere_dev/files/models"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Managed path: /storage/emulated/0/Android/data/me.herethere.pythonhere_dev/files/models/litert-community/SmolLM2-135M-Instruct/SmolLM2_135M_Instruct.litertlm"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Available LiteRT-LM models:"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "["
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'/storage/emulated/0/Android/data/me.herethere.pythonhere_dev/files/models/litert-community/SmolLM2-135M-Instruct/SmolLM2_135M_Instruct.litertlm'"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "%%there\n",
    "from pprint import pprint as pp\n",
    "from ml_here import (\n",
    "    discover_models,\n",
    "    download_hf_model,\n",
    "    model_path,\n",
    "    models_directory,\n",
    "    require_model,\n",
    ")\n",
    "\n",
    "MODEL_REPO = \"litert-community/SmolLM2-135M-Instruct\"\n",
    "MODEL_FILE = \"SmolLM2_135M_Instruct.litertlm\"\n",
    "\n",
    "print(f\"Model directory: {models_directory(create=True)}\")\n",
    "print(f\"Managed path: {model_path(MODEL_REPO, MODEL_FILE)}\")\n",
    "print(\"Available LiteRT-LM models:\")\n",
    "pp(discover_models())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c31e86c",
   "metadata": {},
   "source": [
    "## Download the model\n",
    "\n",
    "`download_hf_model()` downloads the model into its managed path. Downloads\n",
    "can be resumed, and an existing verified file is reused. Because downloading\n",
    "may take time, this cell runs on a worker so the Android UI stays responsive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "6b32c650",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-09-06T10:01:32.255847Z",
     "iopub.status.busy": "2026-09-06T10:01:32.255682Z",
     "iopub.status.idle": "2026-09-06T10:01:41.506963Z",
     "shell.execute_reply": "2026-09-06T10:01:41.505588Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Downloaded model: /storage/emulated/0/Android/data/me.herethere.pythonhere_dev/files/models/litert-community/SmolLM2-135M-Instruct/SmolLM2_135M_Instruct.litertlm\n"
     ]
    }
   ],
   "source": [
    "%%there --worker\n",
    "def report_progress(value):\n",
    "    if value.percent is not None:\n",
    "        print(f\"\\rDownloading: {value.percent:.1f}%\", end=\"\", flush=True)\n",
    "\n",
    "\n",
    "downloaded_path = download_hf_model(\n",
    "    repo_id=MODEL_REPO,\n",
    "    filename=MODEL_FILE,\n",
    "    progress=report_progress,\n",
    ")\n",
    "\n",
    "print(f\"\\nDownloaded model: {downloaded_path}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df8d4e4c",
   "metadata": {},
   "source": [
    "`require_model()` returns the model path if the file is available and raises\n",
    "`FileNotFoundError` otherwise. Discover the device again to confirm that the\n",
    "downloaded model is now included:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "1f537e42",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-09-06T10:01:41.512045Z",
     "iopub.status.busy": "2026-09-06T10:01:41.511640Z",
     "iopub.status.idle": "2026-09-06T10:01:41.855718Z",
     "shell.execute_reply": "2026-09-06T10:01:41.854500Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Required model: /storage/emulated/0/Android/data/me.herethere.pythonhere_dev/files/models/litert-community/SmolLM2-135M-Instruct/SmolLM2_135M_Instruct.litertlm"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Available LiteRT-LM models:"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "["
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'/storage/emulated/0/Android/data/me.herethere.pythonhere_dev/files/models/litert-community/SmolLM2-135M-Instruct/SmolLM2_135M_Instruct.litertlm'"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "%%there\n",
    "model_file = require_model(MODEL_REPO, MODEL_FILE)\n",
    "print(f\"Required model: {model_file}\")\n",
    "print(\"Available LiteRT-LM models:\")\n",
    "pp(discover_models())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a98a5cc9",
   "metadata": {},
   "source": [
    "## Load the model\n",
    "\n",
    "Model loading and inference can block the app's UI thread, so they also run\n",
    "with `%%there --worker`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "6895b9c4",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-09-06T10:01:41.860548Z",
     "iopub.status.busy": "2026-09-06T10:01:41.860169Z",
     "iopub.status.idle": "2026-09-06T10:01:43.107989Z",
     "shell.execute_reply": "2026-09-06T10:01:43.106791Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Engine(model_path='/storage/emulated/0/Android/data/me.herethere.pythonhere_dev/files/models/litert-community/SmolLM2-135M-Instruct/SmolLM2_135M_Instruct.litertlm',\n",
      "       backend=CPU(thread_count=None),\n",
      "       max_num_tokens=None,\n",
      "       max_num_images=None,\n",
      "       cache_dir=None,\n",
      "       vision_backend=None,\n",
      "       audio_backend=None,\n",
      "       enable_speculative_decoding=None,\n",
      "       lora_rank_config=None,\n",
      "       activation_data_type=None,\n",
      "       use_ringbuffers_local_attention=None,\n",
      "       enable_ynnpack=False)\n"
     ]
    }
   ],
   "source": [
    "%%there --worker\n",
    "import litert_lm\n",
    "\n",
    "engine = litert_lm.Engine(model_file)\n",
    "pp(engine)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66c741b8",
   "metadata": {},
   "source": [
    "## Generate text\n",
    "\n",
    "Create a conversation and send a prompt. The model runs locally in the\n",
    "PythonHere app, and only the generated response is returned to Jupyter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "2b3e7ada",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-09-06T10:01:43.113043Z",
     "iopub.status.busy": "2026-09-06T10:01:43.112649Z",
     "iopub.status.idle": "2026-09-06T10:01:54.047902Z",
     "shell.execute_reply": "2026-09-06T10:01:54.046683Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('The Kivy framework is a popular programming language and framework for '\n",
      " 'building interactive and dynamic web applications. It is a cross-platform '\n",
      " 'framework that allows developers to create web applications with a wide '\n",
      " 'range of features, including web servers, databases, and APIs. Kivy is a '\n",
      " 'cross-platform framework that allows developers to build web applications '\n",
      " 'with a wide range of features, including web servers, databases, and APIs. '\n",
      " 'It is a popular choice among developers for building web applications, as it '\n",
      " 'provides a wide range of features and is easy to use.')\n"
     ]
    }
   ],
   "source": [
    "%%there --worker\n",
    "with engine.create_conversation(\n",
    "    max_output_tokens=128,\n",
    ") as conversation:\n",
    "    response = conversation.send_message(\n",
    "        \"What is the Kivy framework? Answer in one short paragraph.\"\n",
    "    )\n",
    "    pp(response[\"content\"][0][\"text\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c0e930d",
   "metadata": {},
   "source": [
    "## Clean up\n",
    "\n",
    "Close the engine to release its native resources when it is no longer needed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "ea04089f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-09-06T10:01:54.053270Z",
     "iopub.status.busy": "2026-09-06T10:01:54.052849Z",
     "iopub.status.idle": "2026-09-06T10:01:54.472499Z",
     "shell.execute_reply": "2026-09-06T10:01:54.471134Z"
    }
   },
   "outputs": [],
   "source": [
    "%%there --worker\n",
    "engine.close()"
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "default_lexer": "ipython3"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "there-ipython3",
   "version": "3.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}