Evaluating Models

Evaluator predicts and scores one or more models against a dataset. It works the same way whether a model is registered (loaded locally), deployed (predicted remotely), or a bare in-memory model instance you never registered at all.

This notebook has three sections: segmentation (comparing multiple models), classification (a single-label model) and detection (comparing multiple models).

Segmentation

Setup

[ ]:

from datamint import Api, build_dataset

PROJECT_NAME = "Evaluation_Segmentation_Test"
api = Api()

1. Load the Dataset

Evaluator is built from a fixed dataset, set once at construction and reused by every .evaluate() call.

[9]:

from datamint.evaluate import Evaluator

project = api.projects.get_by_name(PROJECT_NAME)
dataset = build_dataset(project, allow_external_annotations = True)

# lets get the test split of the dataset
test_ds = dataset.split(use_project_splits=True)['test']
print(test_ds)

# getting only 5 samples from the test split
test_ds = test_ds.subset(range(5))
print(test_ds)

evaluator = Evaluator(dataset=test_ds)

INFO:datamint.api.client:Derived MLflow server URL 'https://mlflow.datamint.io:443' from API URL 'https://api.datamint.io'
INFO:datamint.api.client:Derived MLflow server URL 'https://mlflow.datamint.io:443' from API URL 'https://api.datamint.io'
INFO:datamint.dataset.base:Allowing external segmentation label 'benign' not in project specs.
INFO:datamint.dataset.base:Allowing external segmentation label 'malignant' not in project specs.

ImageDataset
  Project: Evaluation_Segmentation_Test
  Number of datapoints: 117
  Split: test
  Split source: project_api
  Split as of: 2026-09-22T13:47:24.762243Z
ImageDataset
  Project: Evaluation_Segmentation_Test
  Number of datapoints: 5
  Split: test
  Split source: project_api
  Split as of: 2026-09-22T13:47:24.762243Z

2. Pick Models to Compare

[10]:

DEEPLAB_MODEL_NAME = "DeepLabV3Plus"
UNETPP_MODEL_NAME = "UNet++_Model"

3. Run the Evaluation

Each entry in models=[...] can be a registered model name, a Model, a ModelVersion, a bare model instance, or a (model, config_dict) pair.

[ ]:

results = evaluator.evaluate(models=[DEEPLAB_MODEL_NAME, UNETPP_MODEL_NAME])
list(results)

4. Reading the Results

Dice and IoU are reported at three levels of granularity on result.scores:

Level	Shape
`per_resource`	`resource_id -> class_name -> {'dice', 'iou'}`
`per_class`	`class_name -> {'dice', 'iou', 'n'}`
`dataset`	`{'dice', 'iou'}` – one overall number per model

A model that doesn’t predict a class present in the ground truth scores 0 for that class/resource pair.

[26]:

# getting the ground truth class names from the test dataset
gt_class_names = {ann.name for anns in test_ds.resource_annotations for ann in anns}
print(gt_class_names)

# printing the evaluation results
for model_name, result in results.items():
    print(f"\n{model_name}")
    print("="*len(model_name))
    print(f" {'overall':14} dice = {result.scores.dataset['dice']:.3f} iou = {result.scores.dataset['iou']:.3f}")
    for class_name, class_scores in result.scores.per_class.items():
        print(f" {class_name:14} dice = {class_scores['dice']:.3f} iou = {class_scores['iou']:.3f}")

{'benign'}

DeepLabV3Plus_v1
================
 overall        dice = 0.815 iou = 0.702
 benign         dice = 0.815 iou = 0.702

UNet++_Model_v1
===============
 overall        dice = 0.313 iou = 0.248
 malignant      dice = 0.000 iou = 0.000
 benign         dice = 0.625 iou = 0.495

5. Hyperparameter Sweeps and Config Overrides

Pass a (model, config_dict) pair to sweep predict-time parameters, e.g. confidence_threshold.

[ ]:

results = evaluator.evaluate(models=[
    DEEPLAB_MODEL_NAME,
    (UNETPP_MODEL_NAME, {"confidence_threshold": 0.6}),
])

6. Saving Predictions and MLflow Logging

save_results=True uploads each model’s predictions back to Datamint as annotations on their resources, whether that model ran locally or through a deployed serving pod.

experiment_name= changes the default name to another one.

[ ]:

results = evaluator.evaluate(
    models=[DEEPLAB_MODEL_NAME, UNETPP_MODEL_NAME],
    save_results=True,
    experiment_name="busi-segmentation-eval",
)

Classification

Setup

[2]:

from datamint import Api, build_dataset

PROJECT_NAME = "ClassificationTest"
api = Api()

7. Load the Dataset

[3]:

from datamint.evaluate import Evaluator

project = api.projects.get_by_name(PROJECT_NAME)
dataset = build_dataset(project, allow_external_annotations=True)

# lets get the test split of the dataset
test_ds = dataset.split(use_project_splits=True)['test']
print(test_ds)

# getting only 5 samples from the test split
test_ds = test_ds.subset(range(5))
print(test_ds)

evaluator = Evaluator(dataset=test_ds)

INFO:datamint.api.client:Derived MLflow server URL 'https://mlflow.datamint.io:443' from API URL 'https://api.datamint.io'
INFO:datamint.api.client:Derived MLflow server URL 'https://mlflow.datamint.io:443' from API URL 'https://api.datamint.io'
INFO:datamint.dataset.base:Allowing external image label '('has_fracture', 'no')' not in project specs.
INFO:datamint.dataset.base:Allowing external image label '('has_fracture', 'yes')' not in project specs.

ImageDataset
  Project: ClassificationTest
  Number of datapoints: 409
  Split: test
  Split source: project_api
  Split as of: 2026-09-22T18:19:08.635846Z
ImageDataset
  Project: ClassificationTest
  Number of datapoints: 5
  Split: test
  Split source: project_api
  Split as of: 2026-09-22T18:19:08.635846Z

8. Pick the Model

[4]:

CLASSIFICATION_MODEL_NAME = "ClassificationTest"

9. Run the Evaluation

[ ]:

results = evaluator.evaluate(models=CLASSIFICATION_MODEL_NAME)
list(results)

10. Reading the Results

Precision/recall/F1 are reported per class, and accuracy/F1 (macro) at the dataset level, on result.scores:

Level	Shape
`per_resource`	`resource_id -> {'true', 'predicted', 'correct'}`
`per_class`	`class_name -> {'precision', 'recall', 'f1', 'n'}`
`dataset`	`{'accuracy', 'f1'}` – one overall number per model

A resource with no ground-truth label, or more than one, is skipped (single-label scoring needs exactly one). A resource the model didn’t predict for (e.g. filtered out by confidence_threshold) counts as incorrect.

[6]:

# getting the ground truth class names from the test dataset
gt_class_names = {ann.value for anns in test_ds.resource_annotations for ann in anns if ann.is_category()}
print(gt_class_names)

# printing the evaluation results
for model_name, result in results.items():
    print(f"\n{model_name}")
    print("="*len(model_name))
    print(f" {'overall':14} accuracy = {result.scores.dataset['accuracy']:.3f} f1 = {result.scores.dataset['f1']:.3f}")
    for class_name, class_scores in result.scores.per_class.items():
        print(f" {class_name:14} precision = {class_scores['precision']:.3f} recall = {class_scores['recall']:.3f} f1 = {class_scores['f1']:.3f}")

{'no'}

ClassificationTest_v6
=====================
 overall        accuracy = 0.750 f1 = 0.429
 no             precision = 1.000 recall = 0.750 f1 = 0.857
 yes            precision = 0.000 recall = 0.000 f1 = 0.000

Detection

Setup

[ ]:

from datamint import Api, build_dataset

PROJECT_NAME = "bccd_detection"
api = Api()

/home/luan/Desktop/Datamint/Codes/datamint-python-api/datamint/env/lib/python3.12/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm
INFO:datamint.api.client:Derived MLflow server URL 'https://mlflow.datamint.io:443' from API URL 'https://api.datamint.io'

11. Load the Dataset

[4]:

from datamint.evaluate import Evaluator

project = api.projects.get_by_name(PROJECT_NAME)
dataset = build_dataset(project, allow_external_annotations=True)

# lets get the test split of the dataset
test_ds = dataset.split(use_project_splits=True)['test']
print(test_ds)

# getting only 5 samples from the test split
test_ds = test_ds.subset(range(5))
print(test_ds)

evaluator = Evaluator(dataset=test_ds)

INFO:datamint.api.client:Derived MLflow server URL 'https://mlflow.datamint.io:443' from API URL 'https://api.datamint.io'
INFO:datamint.api.client:Derived MLflow server URL 'https://mlflow.datamint.io:443' from API URL 'https://api.datamint.io'

ImageDataset
  Project: bccd_detection
  Number of datapoints: 56
  Split: test
  Split source: project_api
  Split as of: 2026-09-24T16:30:38.508750Z
ImageDataset
  Project: bccd_detection
  Number of datapoints: 5
  Split: test
  Split source: project_api
  Split as of: 2026-09-24T16:30:38.508750Z

12. Pick Models to Compare

[7]:

DETECTION_MODEL_A_NAME = "yolox_s_20_epochs"
DETECTION_MODEL_B_NAME = "yolox_s_10_epochs"

13. Run the Evaluation

[ ]:

results = evaluator.evaluate(models=[DETECTION_MODEL_A_NAME, DETECTION_MODEL_B_NAME])
list(results)

14. Reading the Results

Level	Shape
`per_resource`	`resource_id -> {'tp', 'fp', 'fn', 'precision', 'recall'}` at IoU 0.5
`per_class`	`class_name -> {'ap50', 'ap50_95', 'precision', 'recall', 'f1', 'n'}`
`dataset`	`{'map50', 'map50_95'}` – one overall number per model

A predicted box is matched to a ground-truth box of the same class, most confident prediction first. A class present in the ground truth but never predicted scores AP 0. A class the model predicts but that never appears in the ground truth is left out of mAP, and is still listed in per_class with n = 0 and only precision.

[9]:

# getting the ground truth class names from the test dataset
gt_class_names = {ann.identifier for anns in test_ds.resource_annotations for ann in anns if ann.annotation_type == 'square'}
print(gt_class_names)

# printing the evaluation results
for model_name, result in results.items():
    print(f"\n{model_name}")
    print("="*len(model_name))
    print(f" {'overall':14} mAP50 = {result.scores.dataset['map50']:.3f} mAP50:95 = {result.scores.dataset['map50_95']:.3f}")
    for class_name, class_scores in result.scores.per_class.items():
        if class_scores['n'] == 0:
            print(f" {class_name:14} precision = {class_scores['precision']:.3f} (not in ground truth)")
            continue
        print(f" {class_name:14} ap50 = {class_scores['ap50']:.3f} ap50_95 = {class_scores['ap50_95']:.3f} precision = {class_scores['precision']:.3f} recall = {class_scores['recall']:.3f}")

{'Platelets', 'RBC', 'WBC'}

yolox_s_20_epochs_v1
====================
 overall        mAP50 = 0.356 mAP50:95 = 0.140
 Platelets      ap50 = 0.492 ap50_95 = 0.211 precision = 0.636 recall = 0.583
 RBC            ap50 = 0.175 ap50_95 = 0.079 precision = 0.449 recall = 0.331
 WBC            ap50 = 0.402 ap50_95 = 0.129 precision = 0.800 recall = 0.444

yolox_s_10_epochs_v1
====================
 overall        mAP50 = 0.147 mAP50:95 = 0.052
 Platelets      ap50 = 0.000 ap50_95 = 0.000 precision = 0.000 recall = 0.000
 RBC            ap50 = 0.214 ap50_95 = 0.102 precision = 0.609 recall = 0.322
 WBC            ap50 = 0.228 ap50_95 = 0.055 precision = 0.333 recall = 0.333