{ "cells": [ { "cell_type": "markdown", "id": "intro", "metadata": {}, "source": [ "# Running Predictions on Images, Volumes, and Videos\n", "\n", "You trained a 2D segmentation model on plain PNG images.\n", "But what happens when someone sends a CT scan or a video clip to the server?\n", "\n", "Datamint handles this automatically. If your model implements `predict_image`,\n", "the library unlocks all of these modes without any extra code from you:\n", "\n", "| Mode | Input | What happens |\n", "|---|---|---|\n", "| `image` | PNG / JPEG | your model receives the image directly |\n", "| `slice` | 3D volume | one axial slice is extracted → your model receives it as a 2D image |\n", "| `volume` | 3D volume (CT, MRI) | every axial slice is processed → your model is called once per slice |\n", "| `frame` | video | one frame is extracted → your model receives it as a 2D image |\n", "| `all_frames` | video | every frame is processed → your model is called once per frame |\n", "\n", "This notebook shows all of this in action using a real model trained on breast ultrasound images." ] }, { "cell_type": "markdown", "id": "setup-header", "metadata": {}, "source": [ "## Setup" ] }, { "cell_type": "code", "execution_count": 2, "id": "setup", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/home/luan/Desktop/Datamint/Codes/datamint-python-api/datamint/env/lib/python3.12/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n", " from .autonotebook import tqdm as notebook_tqdm\n" ] }, { "data": { "text/html": [ "
\n", "\n", " \n", "
\n", "
Entity
\n", "
\n", "

Project

\n", "
\n", "
\n", "\n", " \n", "
\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Name\n", " deeplabv3plus_Segmentation_Tutorial\n", "
Created At\n", " 2026-06-01T12:48:44.168Z\n", "
Created By\n", " lucasmello.research@gmail.com\n", "
Archived\n", " False\n", "
Resource Count\n", " 780\n", "
\n", "
\n", "\n", "
" ], "text/plain": [ "Project(id='6f0c4ccb-8dda-41c1-be60-144e4953270c', name='deeplabv3plus_Segmentation_Tutorial', created_at='2026-06-01T12:48:44.168Z', created_by='lucasmello.research@gmail.com', dataset_id='9c07a85d-7985-44b3-a46d-404be963024d', archived=False, resource_count=780, description='', most_recent_experiment=None, annotators=[], is_annotation_only=False, is_validation_reader_only=False, reader_study_count=0, can_annotate=False, can_review=False, can_validate=False, is_contributor_only=False)" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import datamint.mlflow as datamint_mlflow\n", "from datamint import Api\n", "\n", "PROJECT_NAME = 'deeplabv3plus_Segmentation_Tutorial'\n", "MODEL_NAME = 'deeplabv3plus_Segmentation_Tutorial'\n", "\n", "api = Api()\n", "datamint_mlflow.set_project(PROJECT_NAME)" ] }, { "cell_type": "markdown", "id": "load-header", "metadata": {}, "source": [ "## 1. Load the Model\n", "\n", "We load the model from the MLflow registry. No deployment needed — inference runs locally." ] }, { "cell_type": "code", "execution_count": 3, "id": "load-model", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Downloading artifacts: 100%|██████████| 6/6 [01:04<00:00, 10.72s/it]\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Model : DeepLabV3PlusModule\n", "Task type : TaskType.IMAGE_SEGMENTATION\n", "Modes : ['image', 'frame', 'all_frames', 'slice', 'volume']\n" ] } ], "source": [ "from datamint.mlflow import flavors as datamint_flavor\n", "\n", "model = datamint_flavor.load_model(f'models:/{MODEL_NAME}/latest')\n", "\n", "print('Model :', type(model).__name__)\n", "print('Task type :', model.task_type)\n", "print('Modes :', model.get_supported_modes())" ] }, { "cell_type": "markdown", "id": "modes-note", "metadata": {}, "source": [ "The model source code only has `predict_image`.\n", "The library automatically adds the other four modes — you should see something like:\n", "```\n", "Modes: ['image', 'frame', 'all_frames', 'slice', 'volume']\n", "```" ] }, { "cell_type": "markdown", "id": "image-header", "metadata": {}, "source": [ "## 2. Predict on a Plain Image\n", "\n", "This is the native case. The model receives a real PNG from the project and `predict_image` is called directly." ] }, { "cell_type": "code", "execution_count": 4, "id": "fetch-image", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "normal (93).png ImageResource\n", "normal (94).png ImageResource\n", "normal (96).png ImageResource\n" ] } ], "source": [ "resources = list(api.resources.get_list(project_name=PROJECT_NAME, limit=3))\n", "\n", "for r in resources:\n", " print(f'{r.filename:<35} {type(r).__name__}')" ] }, { "cell_type": "code", "execution_count": 5, "id": "image-predict", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Resource : normal (93).png\n", "Annotations: 1\n", " class='benign' mask shape=(578, 770)\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "/home/luan/Desktop/Datamint/Codes/datamint-python-api/datamint/env/lib/python3.12/site-packages/albumentations/core/transforms_interface.py:310: UserWarning: The image is already an RGB.\n", " target_function(ensure_contiguous_output(arg), **params),\n" ] } ], "source": [ "image_resource = resources[0]\n", "\n", "predictions = model.predict([image_resource], params={'mode': 'image'})\n", "\n", "print('Resource :', image_resource.filename)\n", "print('Annotations:', len(predictions[0]))\n", "for ann in predictions[0]:\n", " print(f' class={ann.name!r} mask shape={ann.mask.shape}')" ] }, { "cell_type": "markdown", "id": "volume-header", "metadata": {}, "source": [ "## 3. Predict on a 3D Volume\n", "\n", "Same model, different input. The `volume` mode splits the NIfTI into axial slices and\n", "calls `predict_image` on each one. The results from all slices are merged and returned.\n", "\n", "We create a small synthetic volume here to keep the notebook self-contained.\n", "In production this would be a real CT scan or MRI fetched from the project." ] }, { "cell_type": "code", "execution_count": 6, "id": "create-volume", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Is volume: True\n" ] } ], "source": [ "import os\n", "import tempfile\n", "\n", "import nibabel as nib\n", "import numpy as np\n", "\n", "from datamint.entities.resource import LocalResource\n", "\n", "# 64 x 64 x 10 volume — 10 axial slices\n", "data = (np.random.rand(64, 64, 10) * 1000).astype(np.int16)\n", "nib_img = nib.Nifti1Image(data, affine=np.eye(4))\n", "\n", "tmp = tempfile.NamedTemporaryFile(suffix='.nii', delete=False)\n", "nib.save(nib_img, tmp.name)\n", "\n", "volume_resource = LocalResource(\n", " raw_data=open(tmp.name, 'rb').read(),\n", " filename='synthetic_volume.nii',\n", ")\n", "os.unlink(tmp.name)\n", "\n", "print('Is volume:', volume_resource.is_volume())" ] }, { "cell_type": "code", "execution_count": 7, "id": "volume-predict", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Total annotations across all slices: 0\n" ] } ], "source": [ "# mode='volume' → the bridge slices the volume and calls predict_image once per slice\n", "volume_predictions = model.predict([volume_resource], params={'mode': 'volume'})\n", "\n", "print('Total annotations across all slices:', len(volume_predictions[0]))" ] }, { "cell_type": "code", "execution_count": 8, "id": "slice-predict", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Annotations for slice 4: 0\n" ] } ], "source": [ "# mode='slice' → run on a single specific axial slice\n", "slice_predictions = model.predict(\n", " [volume_resource],\n", " params={'mode': 'slice', 'slice_index': 4, 'axis': 'axial'},\n", ")\n", "\n", "print('Annotations for slice 4:', len(slice_predictions[0]))" ] }, { "cell_type": "markdown", "id": "volume-note", "metadata": {}, "source": [ "> **Note:** each slice is processed independently — results are 2D annotations per slice,\n", "> not a merged 3D segmentation. If you need true 3D inference (e.g. NNUNet),\n", "> you implement `predict_volume` directly and the bridge is bypassed entirely." ] }, { "cell_type": "markdown", "id": "video-header", "metadata": {}, "source": [ "## 4. Predict on a Video\n", "\n", "Same idea as volumes but across the time axis. Each frame is extracted and sent to `predict_image`.\n", "\n", "This requires a `VideoResource` fetched from the API (it carries the frame count needed to iterate).\n", "The example below shows what the code looks like — run it once you have a video in your project." ] }, { "cell_type": "code", "execution_count": null, "id": "video-predict", "metadata": {}, "outputs": [], "source": [ "# Fetch a video resource from the project\n", "# video_resources = list(api.resources.get_list(\n", "# project_name=PROJECT_NAME,\n", "# mimetype_prefix='video/',\n", "# limit=1,\n", "# ))\n", "# video_resource = video_resources[0]\n", "#\n", "# # Run on every frame\n", "# predictions = model.predict([video_resource], params={'mode': 'all_frames'})\n", "# print('Annotations across all frames:', len(predictions[0]))\n", "#\n", "# # Or run on one specific frame\n", "# predictions = model.predict([video_resource], params={'mode': 'frame', 'frame_index': 0})\n", "# print('Annotations for frame 0:', len(predictions[0]))\n", "\n", "print('Uncomment the cells above once you have a video resource in the project.')" ] }, { "cell_type": "markdown", "id": "deployed-header", "metadata": {}, "source": [ "## 5. How This Works When the Model Is Deployed\n", "\n", "Everything above runs the same way on the Datamint server.\n", "When a resource is sent to a deployed model, the server calls `.predict()` with the appropriate mode\n", "— the bridge logic is the same whether you are running locally or on the server.\n", "\n", "To deploy this model, see `01_deploy_registered_model.ipynb`." ] } ], "metadata": { "kernelspec": { "display_name": "env", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.3" } }, "nbformat": 4, "nbformat_minor": 5 }