Validating a Model Before Deployment

Deploying a model just to discover it produces wrong outputs is slow and wasteful. validate_model lets you catch problems locally — before anything is pushed to the registry.

It runs two tiers of checks:

Tier	What it checks	Needs data?
Metadata	task_type, annotation_specs, supported_modes	No
Inference	predict() runs, output structure, label consistency	Yes

This notebook uses the bccd_detection project (blood-cell detection with YOLOX) as a concrete example.

Setup

from datamint import Api, validate_model
from datamint.dataset import build_dataset
from datamint.mlflow.flavors import datamint_flavor

PROJECT_NAME = "bccd_detection"
MODEL_NAME = PROJECT_NAME

api = Api()
proj = api.projects.create(
    name=PROJECT_NAME,
    description="Blood cell detection tutorial using YOLOX",
    exists_ok=True,
)

1. Load the Dataset

validate_model pulls sample resources directly from a DatamintBaseDataset. No extra setup is needed — just pass dataset= and it handles the rest.

dataset = build_dataset(project=PROJECT_NAME, allow_external_annotations=True)

print(f"Resources : {len(dataset.resources)}")
print(f"Box labels: {dataset.box_labels_set}")

2. Load the Registered Model

Load any registered model from MLflow using its URI.

model = datamint_flavor.load_model(f"models:/{MODEL_NAME}/latest")

print(f"Model      : {model.__class__.__name__}")
print(f"task_type  : {model.task_type}")
print(f"annotation_specs: {model.annotation_specs}")

Downloading artifacts: 100%|██████████| 6/6 [00:27<00:00,  4.66s/it]

Model      : YOLOXModule
task_type  : TaskType.OBJECT_DETECTION
annotation_specs: [AnnotationSpec(type=<AnnotationType.SQUARE: 'square'>, scope='image', required=False, identifier='Platelets'), AnnotationSpec(type=<AnnotationType.SQUARE: 'square'>, scope='image', required=False, identifier='RBC'), AnnotationSpec(type=<AnnotationType.SQUARE: 'square'>, scope='image', required=False, identifier='WBC')]

3. Run Validation

validate_model accepts either a dataset or a list of resources directly.

• dataset — pulls n_samples resources automatically

• sample_input — pass a list[Resource] directly

report = validate_model(model, dataset=dataset, n_samples=2)
print(report)

4. Reading the Report

Each line maps to one check:

Marker	Meaning
[v]	Check passed, ready to deploy.
[!]	Warning — model still deployable, but worth investigating
[x]	Error — model should not be deployed as-is

5. Passing Resources Directly

If you already have specific resources you want to test against — for example, a known edge case — pass them via sample_input instead of a dataset.

samples = list(dataset.resources[:3])

report = validate_model(model, sample_input=samples)
print(report)