{ "cells": [ { "cell_type": "markdown", "id": "f7279534", "metadata": {}, "source": [ "# LiteRT multimodal lab\n", "\n", "Companion notebook for the video [*Run multimodal AI on Android with Python and LiteRT*](https://youtu.be/w_RZ0Y29A7I).\n", "\n", "## Connect to PythonHere" ] }, { "cell_type": "code", "execution_count": 2, "id": "21ec1741", "metadata": {}, "outputs": [], "source": [ "%load_ext pythonhere\n", "%connect-there" ] }, { "cell_type": "markdown", "id": "3ab7f64a", "metadata": {}, "source": [ "## Show download progress" ] }, { "cell_type": "code", "execution_count": 3, "id": "ba323d6f", "metadata": {}, "outputs": [], "source": [ "%%there kv\n", ":\n", " size_hint: .82, .82\n", " pos_hint: {'center_x': .5, 'center_y': .5}\n", "\n", " canvas:\n", " Color:\n", " rgba: .45, .52, .50, .16\n", "\n", " Line:\n", " width: dp(5)\n", " circle:\n", " self.center_x, \\\n", " self.center_y, \\\n", " min(self.width, self.height) * .34, \\\n", " 0, \\\n", " 360\n", "\n", " Color:\n", " rgba: .30, .40, .37, .95 if root.value > 0 else 0\n", "\n", " Line:\n", " width: dp(10) if root.value >= 100 else dp(9)\n", " circle:\n", " self.center_x, \\\n", " self.center_y, \\\n", " min(self.width, self.height) * .34, \\\n", " 105, \\\n", " 105 + 360 * min(max(root.value, 0), 100) / 100\n", "\n", " Label:\n", " text: \"Ready\" if root.value >= 100 else f\"{root.value:3.0f}%\"\n", " font_size: min(root.width, root.height) * (.11 if root.value >= 100 else .13)\n", " color: .25, .32, .30, .95\n", " center: root.center" ] }, { "cell_type": "code", "execution_count": 4, "id": "bf7ff167", "metadata": {}, "outputs": [], "source": [ "%%there\n", "from kivy.properties import NumericProperty\n", "from kivy.uix.widget import Widget\n", "\n", "\n", "class DownloadProgress(Widget):\n", " value = NumericProperty(0)\n", "\n", "root.clear_widgets()\n", "progress = DownloadProgress()\n", "root.add_widget(progress)" ] }, { "cell_type": "markdown", "id": "eac33758", "metadata": {}, "source": [ "## Download the model" ] }, { "cell_type": "code", "execution_count": 5, "id": "e942af86", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "'/storage/emulated/0/Android/data/me.herethere.pythonhere_dev/files/models/litert-community/SmolVLM2-500M/SmolVLM2-500M.litertlm'\n" ] } ], "source": [ "%%there --worker\n", "from pprint import pprint as pp\n", "from kivy.clock import mainthread\n", "from ml_here import download_hf_model, require_model\n", "\n", "MODEL_REPO = \"litert-community/SmolVLM2-500M\"\n", "MODEL_FILE = \"SmolVLM2-500M.litertlm\"\n", "\n", "\n", "@mainthread\n", "def update_progress(value):\n", " progress.value = value.percent or 0.0\n", "\n", "\n", "download_hf_model(\n", " repo_id=MODEL_REPO,\n", " filename=MODEL_FILE,\n", " progress=update_progress,\n", ")\n", "\n", "pp(require_model(MODEL_REPO, MODEL_FILE))" ] }, { "cell_type": "markdown", "id": "cf0a0277", "metadata": {}, "source": [ "## Load the model" ] }, { "cell_type": "code", "execution_count": 6, "id": "bfe36343", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Engine(model_path='/storage/emulated/0/Android/data/me.herethere.pythonhere_dev/files/models/litert-community/SmolVLM2-500M/SmolVLM2-500M.litertlm',\n", " backend=GPU(gpu_decode_steps_per_sync=None),\n", " max_num_tokens=None,\n", " max_num_images=None,\n", " cache_dir=None,\n", " vision_backend=GPU(gpu_decode_steps_per_sync=None),\n", " audio_backend=None,\n", " enable_speculative_decoding=None,\n", " lora_rank_config=None,\n", " activation_data_type=None,\n", " use_ringbuffers_local_attention=None,\n", " enable_ynnpack=False)\n" ] } ], "source": [ "%%there --worker\n", "import litert_lm\n", "engine = litert_lm.Engine(\n", " require_model(MODEL_REPO, MODEL_FILE),\n", " backend=litert_lm.Backend.GPU(),\n", " vision_backend=litert_lm.Backend.GPU(),\n", " enable_benchmark=True,\n", ")\n", "pp(engine)" ] }, { "cell_type": "markdown", "id": "7cf15932", "metadata": {}, "source": [ "## Generate text\n", "\n", "### Create a streaming output view" ] }, { "cell_type": "code", "execution_count": 7, "id": "3b5756b9", "metadata": {}, "outputs": [], "source": [ "%%there kv\n", ":\n", " size_hint: .88, .7\n", " pos_hint: {'center_x': .5, 'center_y': .5}\n", "\n", " Label:\n", " markup: True\n", " text: root.text\n", " font_size: dp(21)\n", " halign: \"left\"\n", " valign: \"top\"\n", " text_size: root.width, None\n", " pos: root.pos\n", " size: root.size" ] }, { "cell_type": "code", "execution_count": 8, "id": "257b15b9", "metadata": {}, "outputs": [], "source": [ "%%there\n", "from kivy.properties import StringProperty\n", "from kivy.uix.widget import Widget\n", "from kivy.clock import mainthread\n", "\n", "\n", "class ModelOutput(Widget):\n", " text = StringProperty(\"\")\n", " committed = StringProperty(\"\")\n", " newest = StringProperty(\"\")\n", "\n", " @mainthread\n", " def append_chunk(self, chunk):\n", " self.committed += self.newest\n", " self.newest = chunk\n", " self.text = (\n", " f\"[color=#E8EEF3]{self.committed}[/color]\"\n", " f\"[color=#FFD343][b]{self.newest}[/b][/color]\"\n", " )\n", "\n", "root.clear_widgets()\n", "output = ModelOutput()\n", "root.add_widget(output)" ] }, { "cell_type": "markdown", "id": "e0ec5e21", "metadata": {}, "source": [ "### Ask a text question" ] }, { "cell_type": "code", "execution_count": 9, "id": "21e89877", "metadata": {}, "outputs": [], "source": [ "%%there --worker\n", "\n", "sampler = litert_lm.SamplerConfig(\n", " temperature=0.7,\n", " top_p=0.9,\n", " top_k=40,\n", ")\n", "\n", "with engine.create_conversation(\n", " sampler_config=sampler,\n", " max_output_tokens=128,\n", ") as conversation:\n", " for chunk in conversation.send_message_async(\n", " \"What is on-device AI? Give a direct answer in one short paragraph.\"\n", " ):\n", " output.append_chunk(chunk[\"content\"][0][\"text\"])\n", " info = conversation.get_benchmark_info()\n", " output.append_chunk(\"\")" ] }, { "cell_type": "markdown", "id": "ddbf6546", "metadata": {}, "source": [ "### Show benchmark results" ] }, { "cell_type": "code", "execution_count": 10, "id": "866140f5", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "\tBenchmark info for the conversation\n", "Init: 8.35s\n", "TTFT: 1.24s\n", "Prefill: 23 tokens @ 19.8 tok/s\n", "Decode: 84 tokens @ 11.9 tok/s\n" ] } ], "source": [ "%%there\n", "print(\n", " \"\\n\\tBenchmark info for the conversation\\n\"\n", " f\"Init: {info.init_time_in_second:.2f}s\\n\"\n", " f\"TTFT: {info.time_to_first_token_in_second:.2f}s\\n\"\n", " f\"Prefill: {info.last_prefill_token_count} tokens \"\n", " f\"@ {info.last_prefill_tokens_per_second:.1f} tok/s\\n\"\n", " f\"Decode: {info.last_decode_token_count} tokens \"\n", " f\"@ {info.last_decode_tokens_per_second:.1f} tok/s\"\n", ")" ] }, { "cell_type": "markdown", "id": "e382e06e", "metadata": {}, "source": [ "## Describe a camera photo\n", "\n", "### Request camera permission" ] }, { "cell_type": "code", "execution_count": 11, "id": "15092a72", "metadata": {}, "outputs": [], "source": [ "%%there\n", "from threading import Event\n", "from android.permissions import Permission, request_permission\n", "\n", "done = Event()\n", "request_permission(Permission.CAMERA, lambda *_: done.set())\n", "done.wait()" ] }, { "cell_type": "markdown", "id": "065afc99", "metadata": {}, "source": [ "### Preview and capture a photo" ] }, { "cell_type": "code", "execution_count": 12, "id": "2e6e79c8", "metadata": {}, "outputs": [], "source": [ "%%there kv\n", "AnchorLayout:\n", " anchor_x: \"center\"\n", " anchor_y: \"center\"\n", "\n", " Camera:\n", " id: camera\n", " play: True\n", " resolution: (640, 480)\n", " fit_mode: \"contain\"\n", "\n", " size_hint: None, None\n", " height: min(root.width, root.height * 3 / 4)\n", " width: self.height * 4 / 3\n", "\n", " canvas.before:\n", " PushMatrix\n", " Rotate:\n", " angle: -90\n", " origin: self.center\n", "\n", " canvas.after:\n", " PopMatrix" ] }, { "cell_type": "code", "execution_count": 13, "id": "b0979045", "metadata": {}, "outputs": [], "source": [ "%%there\n", "camera = root.ids.camera\n", "camera.play = False" ] }, { "cell_type": "code", "execution_count": 14, "id": "aae6232d", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Photo: /data/data/me.herethere.pythonhere_dev/files/upload/from_camera.jpg\n", "Image: 480 × 640, RGB, 38.2 KiB\n" ] } ], "source": [ "%%there\n", "from os.path import abspath, getsize\n", "from PIL import Image\n", "\n", "texture = camera.texture\n", "image = Image.frombytes(\"RGBA\", texture.size, texture.pixels)\n", "\n", "# Kivy textures use a bottom-left origin.\n", "image = image.transpose(Image.Transpose.FLIP_TOP_BOTTOM)\n", "\n", "# Match the rotation applied to the Android camera preview.\n", "image = image.rotate(-90, expand=True).convert(\"RGB\")\n", "\n", "photo_path = abspath(\"from_camera.jpg\")\n", "image.save(photo_path, quality=90)\n", "\n", "print(f\"Photo: {photo_path}\")\n", "print(\n", " f\"Image: {image.width} × {image.height}, \"\n", " f\"{image.mode}, {getsize(photo_path) / 1024:.1f} KiB\"\n", ")" ] }, { "cell_type": "markdown", "id": "c4b9c200", "metadata": {}, "source": [ "### Generate a description" ] }, { "cell_type": "code", "execution_count": 16, "id": "116170e6", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " A mug likely belongs to the maker of Android or Kinder Stills for Samsung of Ireland with the Kitter Still Designs company. Here's a visual guide to understanding what could possibly fit inside the mug. However, this is purely speculative:\n", "\n", "A mug likely fits a teacup. The teacup has a handle, a handle inside it is difficult to determine and may not fit directly with the interior inside. It might fit into what looks similar to a m\n" ] } ], "source": [ "%%there --worker\n", "sampler = litert_lm.SamplerConfig(\n", " temperature=1.5,\n", " top_p=0.9,\n", " top_k=40,\n", ")\n", "\n", "with engine.create_conversation(\n", " sampler_config=sampler,\n", " max_output_tokens=96,\n", ") as conversation:\n", " response = conversation.send_message(\n", " litert_lm.Contents.of(\n", " \"What is it?\",\n", " litert_lm.Content.ImageFile(absolute_path=photo_path),\n", " )\n", " )\n", " print(response[\"content\"][0][\"text\"])" ] }, { "cell_type": "markdown", "id": "d8a2eecb", "metadata": {}, "source": [ "## Explore sampling variability\n", "\n", "### Generate responses with different temperatures and seeds" ] }, { "cell_type": "code", "execution_count": 17, "id": "ce60534f", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Done: 20 runs\n" ] } ], "source": [ "%%there --worker\n", "prompt = (\n", " \"Describe the main object in this image clearly and concisely \"\n", " \"in 2-3 sentences.\"\n", ")\n", "\n", "temperatures = [0.2, 0.5, 0.7, 1.0]\n", "seeds = [1, 7, 42, 123, 999]\n", "\n", "experiment_results = []\n", "\n", "for temperature in temperatures:\n", " for seed in seeds:\n", " sampler = litert_lm.SamplerConfig(\n", " temperature=temperature,\n", " top_p=0.9,\n", " top_k=40,\n", " seed=seed,\n", " )\n", "\n", " with engine.create_conversation(\n", " sampler_config=sampler,\n", " max_output_tokens=96,\n", " ) as conversation:\n", " response = conversation.send_message(\n", " litert_lm.Contents.of(\n", " prompt,\n", " litert_lm.Content.ImageFile(absolute_path=photo_path),\n", " )\n", " )\n", "\n", " info = conversation.get_benchmark_info()\n", "\n", " experiment_results.append({\n", " \"temperature\": temperature,\n", " \"seed\": seed,\n", " \"output\": response[\"content\"][0][\"text\"].strip(),\n", " \"ttft_s\": info.time_to_first_token_in_second,\n", " \"prefill_tokens\": info.last_prefill_token_count,\n", " \"prefill_tok_s\": info.last_prefill_tokens_per_second,\n", " \"decode_tokens\": info.last_decode_token_count,\n", " \"decode_tok_s\": info.last_decode_tokens_per_second,\n", " })\n", "\n", "print(f\"Done: {len(experiment_results)} runs\")" ] }, { "cell_type": "markdown", "id": "14417467", "metadata": {}, "source": [ "### Compare the outputs" ] }, { "cell_type": "code", "execution_count": 18, "id": "0771697f", "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
temperatureseeddecode_tokensoutput
0.2120The main object in the image is a mug with a cartoon penguin and Android mascot.
0.2720The main object in the image is a mug with a cartoon penguin and Android robot design.
0.24235The main object in the image is a mug with a cartoon character design. The mug is clear and has a handle, which is placed on a white surface.
0.212335The main object in the image is a mug with a cartoon character design. The mug has a handle on the right side and is placed on a white surface.
0.299921The main object in the image is a mug with a cartoon penguin and Android character on it.
0.5155The main object in the image is a mug with a cartoon design. The mug features an Android robot and a penguin on it. The robot is green and has a yellow car on its head. The penguin is blue and has a yellow car on its head.
0.5720The main object in the image is a mug with a cartoon penguin and Android robot design.
0.54267The main object in the image is a mug with a cartoon character. The mug is clear and has a handle, which is located on the right side of the mug. The cartoon character is a robot and a penguin. The mug is placed on a white surface, which is likely a counter or a table.
0.512319The main object in the image is a mug with a cartoon penguin and robot design.
0.599920The main object in the image is a mug with a cartoon penguin and Android mascot.
0.7139The main object in the image is a mug with a cartoon drawing of a penguin, a smiling face, and a small blue robot. The mug is placed on a white surface.
0.7746The main object in the image is a mug with a cartoon design. The mug has a handle on the right side and a small cartoon character on it. The character is green with a blue bird on its head.
0.74247The main object in the image is a mug with a cartoon character drawing on it. The mug has a handle and is decorated with a cartoon character, including a blue penguin. The cartoon character is riding a skateboard.
0.712319The main object in the image is a mug with a cartoon penguin and robot drawing.
0.799927The main object in the image is a mug with a cartoon design. The mug features a cartoon robot and penguin characters.
1.0153The main object in the image is a mug with a design of an Android character with a bird and a penguin. The mug has a handle, a lid, and is decorated with a design of an Android character, a bird, and a penguin.
1.0724The main object in the image is a clear mug that has a cartoon penguin and android drawing on it.
1.04218The main object in the image is a mug featuring a cartoon character on it.
1.012395The main object in the image is a mug with an illustration of an android and penguin. The mug is set against a white background, which is commonly used in presentations to keep the focus on the subject without distractions. The illustration is centered, with a small, stylized mug icon in the top right corner, which suggests that the mug is not an actual mug, but rather an example to illustrate or enhance the mug's design or functionality.
1.099937The main object in the image is a mug with a picture of a robot and a penguin. The mug is made of glass and is colored in a light blue color.
\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import pandas as pd\n", "\n", "experiment_results = %there get experiment_results\n", "\n", "df = pd.DataFrame(experiment_results)\n", "\n", "display(\n", " df[\n", " [\n", " \"temperature\",\n", " \"seed\",\n", " \"decode_tokens\",\n", " \"output\",\n", " ]\n", " ].style\n", " .format({\n", " \"temperature\": \"{:.1f}\",\n", " })\n", " .set_properties(\n", " subset=[\"output\"],\n", " **{\n", " \"white-space\": \"pre-wrap\",\n", " \"text-align\": \"left\",\n", " },\n", " )\n", " .hide(axis=\"index\")\n", ")" ] }, { "cell_type": "markdown", "id": "cd652dea", "metadata": {}, "source": [ "## Clean up" ] }, { "cell_type": "code", "execution_count": 19, "id": "9a0a487b", "metadata": {}, "outputs": [], "source": [ "%%there --worker\n", "engine.close()" ] } ], "metadata": { "jupytext": { "default_lexer": "ipython3" }, "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "name": "python", "pygments_lexer": "there-ipython3" } }, "nbformat": 4, "nbformat_minor": 5 }