Running Predictions on Images, Volumes, and Videos
You trained a 2D segmentation model on plain PNG images. But what happens when someone sends a CT scan or a video clip to the server?
Datamint handles this automatically. If your model implements predict_image, the library unlocks all of these modes without any extra code from you:
Mode |
Input |
What happens |
|---|---|---|
|
PNG / JPEG |
your model receives the image directly |
|
3D volume |
one axial slice is extracted → your model receives it as a 2D image |
|
3D volume (CT, MRI) |
every axial slice is processed → your model is called once per slice |
|
video |
one frame is extracted → your model receives it as a 2D image |
|
video |
every frame is processed → your model is called once per frame |
This notebook shows all of this in action using a real model trained on breast ultrasound images.
Setup
[2]:
import datamint.mlflow as datamint_mlflow
from datamint import Api
PROJECT_NAME = 'deeplabv3plus_Segmentation_Tutorial'
MODEL_NAME = 'deeplabv3plus_Segmentation_Tutorial'
api = Api()
datamint_mlflow.set_project(PROJECT_NAME)
/home/luan/Desktop/Datamint/Codes/datamint-python-api/datamint/env/lib/python3.12/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
from .autonotebook import tqdm as notebook_tqdm
[2]:
1. Load the Model
We load the model from the MLflow registry. No deployment needed — inference runs locally.
[3]:
from datamint.mlflow import flavors as datamint_flavor
model = datamint_flavor.load_model(f'models:/{MODEL_NAME}/latest')
print('Model :', type(model).__name__)
print('Task type :', model.task_type)
print('Modes :', model.get_supported_modes())
Downloading artifacts: 100%|██████████| 6/6 [01:04<00:00, 10.72s/it]
Model : DeepLabV3PlusModule
Task type : TaskType.IMAGE_SEGMENTATION
Modes : ['image', 'frame', 'all_frames', 'slice', 'volume']
The model source code only has predict_image. The library automatically adds the other four modes — you should see something like:
Modes: ['image', 'frame', 'all_frames', 'slice', 'volume']
2. Predict on a Plain Image
This is the native case. The model receives a real PNG from the project and predict_image is called directly.
[4]:
resources = list(api.resources.get_list(project_name=PROJECT_NAME, limit=3))
for r in resources:
print(f'{r.filename:<35} {type(r).__name__}')
normal (93).png ImageResource
normal (94).png ImageResource
normal (96).png ImageResource
[5]:
image_resource = resources[0]
predictions = model.predict([image_resource], params={'mode': 'image'})
print('Resource :', image_resource.filename)
print('Annotations:', len(predictions[0]))
for ann in predictions[0]:
print(f' class={ann.name!r} mask shape={ann.mask.shape}')
Resource : normal (93).png
Annotations: 1
class='benign' mask shape=(578, 770)
/home/luan/Desktop/Datamint/Codes/datamint-python-api/datamint/env/lib/python3.12/site-packages/albumentations/core/transforms_interface.py:310: UserWarning: The image is already an RGB.
target_function(ensure_contiguous_output(arg), **params),
3. Predict on a 3D Volume
Same model, different input. The volume mode splits the NIfTI into axial slices and calls predict_image on each one. The results from all slices are merged and returned.
We create a small synthetic volume here to keep the notebook self-contained. In production this would be a real CT scan or MRI fetched from the project.
[6]:
import os
import tempfile
import nibabel as nib
import numpy as np
from datamint.entities.resource import LocalResource
# 64 x 64 x 10 volume — 10 axial slices
data = (np.random.rand(64, 64, 10) * 1000).astype(np.int16)
nib_img = nib.Nifti1Image(data, affine=np.eye(4))
tmp = tempfile.NamedTemporaryFile(suffix='.nii', delete=False)
nib.save(nib_img, tmp.name)
volume_resource = LocalResource(
raw_data=open(tmp.name, 'rb').read(),
filename='synthetic_volume.nii',
)
os.unlink(tmp.name)
print('Is volume:', volume_resource.is_volume())
Is volume: True
[7]:
# mode='volume' → the bridge slices the volume and calls predict_image once per slice
volume_predictions = model.predict([volume_resource], params={'mode': 'volume'})
print('Total annotations across all slices:', len(volume_predictions[0]))
Total annotations across all slices: 0
[8]:
# mode='slice' → run on a single specific axial slice
slice_predictions = model.predict(
[volume_resource],
params={'mode': 'slice', 'slice_index': 4, 'axis': 'axial'},
)
print('Annotations for slice 4:', len(slice_predictions[0]))
Annotations for slice 4: 0
Note: each slice is processed independently — results are 2D annotations per slice, not a merged 3D segmentation. If you need true 3D inference (e.g. NNUNet), you implement
predict_volumedirectly and the bridge is bypassed entirely.
4. Predict on a Video
Same idea as volumes but across the time axis. Each frame is extracted and sent to predict_image.
This requires a VideoResource fetched from the API (it carries the frame count needed to iterate). The example below shows what the code looks like — run it once you have a video in your project.
[ ]:
# Fetch a video resource from the project
# video_resources = list(api.resources.get_list(
# project_name=PROJECT_NAME,
# mimetype_prefix='video/',
# limit=1,
# ))
# video_resource = video_resources[0]
#
# # Run on every frame
# predictions = model.predict([video_resource], params={'mode': 'all_frames'})
# print('Annotations across all frames:', len(predictions[0]))
#
# # Or run on one specific frame
# predictions = model.predict([video_resource], params={'mode': 'frame', 'frame_index': 0})
# print('Annotations for frame 0:', len(predictions[0]))
print('Uncomment the cells above once you have a video resource in the project.')
5. How This Works When the Model Is Deployed
Everything above runs the same way on the Datamint server. When a resource is sent to a deployed model, the server calls .predict() with the appropriate mode — the bridge logic is the same whether you are running locally or on the server.
To deploy this model, see 01_deploy_registered_model.ipynb.