{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "bf1e4b63",
   "metadata": {},
   "source": [
    "# Evaluating Models\n",
    "\n",
    "`Evaluator` predicts and scores one or more models against a dataset. It works the same way whether a model is registered (loaded locally), deployed (predicted remotely), or a bare in-memory model instance you never registered at all.\n",
    "\n",
    "This notebook has three sections: segmentation (comparing multiple models), classification (a single-label model) and detection (comparing multiple models)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99151001",
   "metadata": {},
   "source": [
    "## Segmentation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a943c35",
   "metadata": {},
   "source": [
    "### Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f504dff4",
   "metadata": {},
   "outputs": [],
   "source": [
    "from datamint import Api, build_dataset\n",
    "\n",
    "PROJECT_NAME = \"Evaluation_Segmentation_Test\"\n",
    "api = Api()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9754dbd6",
   "metadata": {},
   "source": [
    "### 1. Load the Dataset\n",
    "\n",
    "`Evaluator` is built from a fixed dataset, set once at construction and reused by every `.evaluate()` call."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "2992c427",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:datamint.api.client:Derived MLflow server URL 'https://mlflow.datamint.io:443' from API URL 'https://api.datamint.io'\n",
      "INFO:datamint.api.client:Derived MLflow server URL 'https://mlflow.datamint.io:443' from API URL 'https://api.datamint.io'\n",
      "INFO:datamint.dataset.base:Allowing external segmentation label 'benign' not in project specs.\n",
      "INFO:datamint.dataset.base:Allowing external segmentation label 'malignant' not in project specs.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ImageDataset\n",
      "  Project: Evaluation_Segmentation_Test\n",
      "  Number of datapoints: 117\n",
      "  Split: test\n",
      "  Split source: project_api\n",
      "  Split as of: 2026-09-22T13:47:24.762243Z\n",
      "ImageDataset\n",
      "  Project: Evaluation_Segmentation_Test\n",
      "  Number of datapoints: 5\n",
      "  Split: test\n",
      "  Split source: project_api\n",
      "  Split as of: 2026-09-22T13:47:24.762243Z\n"
     ]
    }
   ],
   "source": [
    "from datamint.evaluate import Evaluator\n",
    "\n",
    "project = api.projects.get_by_name(PROJECT_NAME)\n",
    "dataset = build_dataset(project, allow_external_annotations = True)\n",
    "\n",
    "# lets get the test split of the dataset\n",
    "test_ds = dataset.split(use_project_splits=True)['test']\n",
    "print(test_ds)\n",
    "\n",
    "# getting only 5 samples from the test split\n",
    "test_ds = test_ds.subset(range(5))\n",
    "print(test_ds)\n",
    "\n",
    "evaluator = Evaluator(dataset=test_ds)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f22d6381",
   "metadata": {},
   "source": [
    "### 2. Pick Models to Compare"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "67fc2a28",
   "metadata": {},
   "outputs": [],
   "source": [
    "DEEPLAB_MODEL_NAME = \"DeepLabV3Plus\"\n",
    "UNETPP_MODEL_NAME = \"UNet++_Model\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5ae5141",
   "metadata": {},
   "source": [
    "### 3. Run the Evaluation\n",
    "\n",
    "Each entry in `models=[...]` can be a registered model name, a `Model`, a `ModelVersion`, a bare model instance, or a `(model, config_dict)` pair. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a1c291e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "results = evaluator.evaluate(models=[DEEPLAB_MODEL_NAME, UNETPP_MODEL_NAME])\n",
    "list(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "692e9c0f",
   "metadata": {},
   "source": [
    "### 4. Reading the Results\n",
    "\n",
    "Dice and IoU are reported at three levels of granularity on `result.scores`:\n",
    "\n",
    "| Level | Shape |\n",
    "|---|---|\n",
    "| `per_resource` | `resource_id -> class_name -> {'dice', 'iou'}` |\n",
    "| `per_class` | `class_name -> {'dice', 'iou', 'n'}` |\n",
    "| `dataset` | `{'dice', 'iou'}` -- one overall number per model |\n",
    "\n",
    "A model that doesn't predict a class present in the ground truth scores 0 for that class/resource pair."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "fb76188b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'benign'}\n",
      "\n",
      "DeepLabV3Plus_v1\n",
      "================\n",
      " overall        dice = 0.815 iou = 0.702\n",
      " benign         dice = 0.815 iou = 0.702\n",
      "\n",
      "UNet++_Model_v1\n",
      "===============\n",
      " overall        dice = 0.313 iou = 0.248\n",
      " malignant      dice = 0.000 iou = 0.000\n",
      " benign         dice = 0.625 iou = 0.495\n"
     ]
    }
   ],
   "source": [
    "# getting the ground truth class names from the test dataset\n",
    "gt_class_names = {ann.name for anns in test_ds.resource_annotations for ann in anns}\n",
    "print(gt_class_names)\n",
    "\n",
    "# printing the evaluation results\n",
    "for model_name, result in results.items():\n",
    "    print(f\"\\n{model_name}\")\n",
    "    print(\"=\"*len(model_name))\n",
    "    print(f\" {'overall':14} dice = {result.scores.dataset['dice']:.3f} iou = {result.scores.dataset['iou']:.3f}\")\n",
    "    for class_name, class_scores in result.scores.per_class.items():\n",
    "        print(f\" {class_name:14} dice = {class_scores['dice']:.3f} iou = {class_scores['iou']:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7116e71f",
   "metadata": {},
   "source": [
    "### 5. Hyperparameter Sweeps and Config Overrides\n",
    "\n",
    "Pass a `(model, config_dict)` pair to sweep predict-time parameters, e.g. `confidence_threshold`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7c4f2974",
   "metadata": {},
   "outputs": [],
   "source": [
    "results = evaluator.evaluate(models=[\n",
    "    DEEPLAB_MODEL_NAME,\n",
    "    (UNETPP_MODEL_NAME, {\"confidence_threshold\": 0.6}),\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54c375e5",
   "metadata": {},
   "source": [
    "### 6. Saving Predictions and MLflow Logging\n",
    "\n",
    "`save_results=True` uploads each model's predictions back to Datamint as annotations on their resources, whether that model ran locally or through a deployed serving pod.\n",
    "\n",
    "`experiment_name=` changes the default name to another one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fb5ed971",
   "metadata": {},
   "outputs": [],
   "source": [
    "results = evaluator.evaluate(\n",
    "    models=[DEEPLAB_MODEL_NAME, UNETPP_MODEL_NAME],\n",
    "    save_results=True,\n",
    "    experiment_name=\"busi-segmentation-eval\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66be877e",
   "metadata": {},
   "source": [
    "## Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79bed9c6",
   "metadata": {},
   "source": [
    "### Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "92c73e97",
   "metadata": {},
   "outputs": [],
   "source": [
    "from datamint import Api, build_dataset\n",
    "\n",
    "PROJECT_NAME = \"ClassificationTest\"\n",
    "api = Api()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50e4edf9",
   "metadata": {},
   "source": [
    "### 7. Load the Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "e52c46b2",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:datamint.api.client:Derived MLflow server URL 'https://mlflow.datamint.io:443' from API URL 'https://api.datamint.io'\n",
      "INFO:datamint.api.client:Derived MLflow server URL 'https://mlflow.datamint.io:443' from API URL 'https://api.datamint.io'\n",
      "INFO:datamint.dataset.base:Allowing external image label '('has_fracture', 'no')' not in project specs.\n",
      "INFO:datamint.dataset.base:Allowing external image label '('has_fracture', 'yes')' not in project specs.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ImageDataset\n",
      "  Project: ClassificationTest\n",
      "  Number of datapoints: 409\n",
      "  Split: test\n",
      "  Split source: project_api\n",
      "  Split as of: 2026-09-22T18:19:08.635846Z\n",
      "ImageDataset\n",
      "  Project: ClassificationTest\n",
      "  Number of datapoints: 5\n",
      "  Split: test\n",
      "  Split source: project_api\n",
      "  Split as of: 2026-09-22T18:19:08.635846Z\n"
     ]
    }
   ],
   "source": [
    "from datamint.evaluate import Evaluator\n",
    "\n",
    "project = api.projects.get_by_name(PROJECT_NAME)\n",
    "dataset = build_dataset(project, allow_external_annotations=True)\n",
    "\n",
    "# lets get the test split of the dataset\n",
    "test_ds = dataset.split(use_project_splits=True)['test']\n",
    "print(test_ds)\n",
    "\n",
    "# getting only 5 samples from the test split\n",
    "test_ds = test_ds.subset(range(5))\n",
    "print(test_ds)\n",
    "\n",
    "evaluator = Evaluator(dataset=test_ds)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21c450a9",
   "metadata": {},
   "source": [
    "### 8. Pick the Model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "5c716920",
   "metadata": {},
   "outputs": [],
   "source": [
    "CLASSIFICATION_MODEL_NAME = \"ClassificationTest\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a704591d",
   "metadata": {},
   "source": [
    "### 9. Run the Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cac7065a",
   "metadata": {},
   "outputs": [],
   "source": [
    "results = evaluator.evaluate(models=CLASSIFICATION_MODEL_NAME)\n",
    "list(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c720124",
   "metadata": {},
   "source": [
    "### 10. Reading the Results\n",
    "\n",
    "Precision/recall/F1 are reported per class, and accuracy/F1 (macro) at the dataset level, on `result.scores`:\n",
    "\n",
    "| Level | Shape |\n",
    "|---|---|\n",
    "| `per_resource` | `resource_id -> {'true', 'predicted', 'correct'}` |\n",
    "| `per_class` | `class_name -> {'precision', 'recall', 'f1', 'n'}` |\n",
    "| `dataset` | `{'accuracy', 'f1'}` -- one overall number per model |\n",
    "\n",
    "A resource with no ground-truth label, or more than one, is skipped (single-label scoring needs exactly one). A resource the model didn't predict for (e.g. filtered out by `confidence_threshold`) counts as incorrect."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "b2e5f1c3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'no'}\n",
      "\n",
      "ClassificationTest_v6\n",
      "=====================\n",
      " overall        accuracy = 0.750 f1 = 0.429\n",
      " no             precision = 1.000 recall = 0.750 f1 = 0.857\n",
      " yes            precision = 0.000 recall = 0.000 f1 = 0.000\n"
     ]
    }
   ],
   "source": [
    "# getting the ground truth class names from the test dataset\n",
    "gt_class_names = {ann.value for anns in test_ds.resource_annotations for ann in anns if ann.is_category()}\n",
    "print(gt_class_names)\n",
    "\n",
    "# printing the evaluation results\n",
    "for model_name, result in results.items():\n",
    "    print(f\"\\n{model_name}\")\n",
    "    print(\"=\"*len(model_name))\n",
    "    print(f\" {'overall':14} accuracy = {result.scores.dataset['accuracy']:.3f} f1 = {result.scores.dataset['f1']:.3f}\")\n",
    "    for class_name, class_scores in result.scores.per_class.items():\n",
    "        print(f\" {class_name:14} precision = {class_scores['precision']:.3f} recall = {class_scores['recall']:.3f} f1 = {class_scores['f1']:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3fc905d",
   "metadata": {},
   "source": [
    "## Detection"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "852c1a91",
   "metadata": {},
   "source": [
    "### Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e06caae2",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/luan/Desktop/Datamint/Codes/datamint-python-api/datamint/env/lib/python3.12/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n",
      "INFO:datamint.api.client:Derived MLflow server URL 'https://mlflow.datamint.io:443' from API URL 'https://api.datamint.io'\n"
     ]
    }
   ],
   "source": [
    "from datamint import Api, build_dataset\n",
    "\n",
    "PROJECT_NAME = \"bccd_detection\"\n",
    "api = Api()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71660837",
   "metadata": {},
   "source": [
    "### 11. Load the Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "9a0a3862",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:datamint.api.client:Derived MLflow server URL 'https://mlflow.datamint.io:443' from API URL 'https://api.datamint.io'\n",
      "INFO:datamint.api.client:Derived MLflow server URL 'https://mlflow.datamint.io:443' from API URL 'https://api.datamint.io'\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ImageDataset\n",
      "  Project: bccd_detection\n",
      "  Number of datapoints: 56\n",
      "  Split: test\n",
      "  Split source: project_api\n",
      "  Split as of: 2026-09-24T16:30:38.508750Z\n",
      "ImageDataset\n",
      "  Project: bccd_detection\n",
      "  Number of datapoints: 5\n",
      "  Split: test\n",
      "  Split source: project_api\n",
      "  Split as of: 2026-09-24T16:30:38.508750Z\n"
     ]
    }
   ],
   "source": [
    "from datamint.evaluate import Evaluator\n",
    "\n",
    "project = api.projects.get_by_name(PROJECT_NAME)\n",
    "dataset = build_dataset(project, allow_external_annotations=True)\n",
    "\n",
    "# lets get the test split of the dataset\n",
    "test_ds = dataset.split(use_project_splits=True)['test']\n",
    "print(test_ds)\n",
    "\n",
    "# getting only 5 samples from the test split\n",
    "test_ds = test_ds.subset(range(5))\n",
    "print(test_ds)\n",
    "\n",
    "evaluator = Evaluator(dataset=test_ds)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c68feae8",
   "metadata": {},
   "source": [
    "### 12. Pick Models to Compare"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "0c06422c",
   "metadata": {},
   "outputs": [],
   "source": [
    "DETECTION_MODEL_A_NAME = \"yolox_s_20_epochs\"\n",
    "DETECTION_MODEL_B_NAME = \"yolox_s_10_epochs\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1de8211c",
   "metadata": {},
   "source": [
    "### 13. Run the Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "55965920",
   "metadata": {},
   "outputs": [],
   "source": [
    "results = evaluator.evaluate(models=[DETECTION_MODEL_A_NAME, DETECTION_MODEL_B_NAME])\n",
    "list(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c983168",
   "metadata": {},
   "source": [
    "### 14. Reading the Results\n",
    "\n",
    "| Level | Shape |\n",
    "|---|---|\n",
    "| `per_resource` | `resource_id -> {'tp', 'fp', 'fn', 'precision', 'recall'}` at IoU 0.5 |\n",
    "| `per_class` | `class_name -> {'ap50', 'ap50_95', 'precision', 'recall', 'f1', 'n'}` |\n",
    "| `dataset` | `{'map50', 'map50_95'}` -- one overall number per model |\n",
    "\n",
    "A predicted box is matched to a ground-truth box of the same class, most confident prediction first. A class present in the ground truth but never predicted scores AP 0. A class the model predicts but that never appears in the ground truth is left out of mAP, and is still listed in `per_class` with `n = 0` and only `precision`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "fbc5dd90",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'Platelets', 'RBC', 'WBC'}\n",
      "\n",
      "yolox_s_20_epochs_v1\n",
      "====================\n",
      " overall        mAP50 = 0.356 mAP50:95 = 0.140\n",
      " Platelets      ap50 = 0.492 ap50_95 = 0.211 precision = 0.636 recall = 0.583\n",
      " RBC            ap50 = 0.175 ap50_95 = 0.079 precision = 0.449 recall = 0.331\n",
      " WBC            ap50 = 0.402 ap50_95 = 0.129 precision = 0.800 recall = 0.444\n",
      "\n",
      "yolox_s_10_epochs_v1\n",
      "====================\n",
      " overall        mAP50 = 0.147 mAP50:95 = 0.052\n",
      " Platelets      ap50 = 0.000 ap50_95 = 0.000 precision = 0.000 recall = 0.000\n",
      " RBC            ap50 = 0.214 ap50_95 = 0.102 precision = 0.609 recall = 0.322\n",
      " WBC            ap50 = 0.228 ap50_95 = 0.055 precision = 0.333 recall = 0.333\n"
     ]
    }
   ],
   "source": [
    "# getting the ground truth class names from the test dataset\n",
    "gt_class_names = {ann.identifier for anns in test_ds.resource_annotations for ann in anns if ann.annotation_type == 'square'}\n",
    "print(gt_class_names)\n",
    "\n",
    "# printing the evaluation results\n",
    "for model_name, result in results.items():\n",
    "    print(f\"\\n{model_name}\")\n",
    "    print(\"=\"*len(model_name))\n",
    "    print(f\" {'overall':14} mAP50 = {result.scores.dataset['map50']:.3f} mAP50:95 = {result.scores.dataset['map50_95']:.3f}\")\n",
    "    for class_name, class_scores in result.scores.per_class.items():\n",
    "        if class_scores['n'] == 0:\n",
    "            print(f\" {class_name:14} precision = {class_scores['precision']:.3f} (not in ground truth)\")\n",
    "            continue\n",
    "        print(f\" {class_name:14} ap50 = {class_scores['ap50']:.3f} ap50_95 = {class_scores['ap50_95']:.3f} precision = {class_scores['precision']:.3f} recall = {class_scores['recall']:.3f}\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "env",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}