{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a1b2c3d4",
   "metadata": {},
   "source": [
    "# Validating a Model Before Deployment\n",
    "\n",
    "Deploying a model just to discover it produces wrong outputs is slow and wasteful.\n",
    "`validate_model` lets you catch problems locally — before anything is pushed to the registry.\n",
    "\n",
    "It runs two tiers of checks:\n",
    "\n",
    "| Tier | What it checks | Needs data? |\n",
    "|------|---------------|-------------|\n",
    "| Metadata | `task_type`, `annotation_specs`, `supported_modes` | No |\n",
    "| Inference | `predict()` runs, output structure, label consistency | Yes |\n",
    "\n",
    "This notebook uses the `bccd_detection` project (blood-cell detection with YOLOX) as a concrete example."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2c3d4e5",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "c3d4e5f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "from datamint import Api, validate_model\n",
    "from datamint.dataset import build_dataset\n",
    "from datamint.mlflow.flavors import datamint_flavor\n",
    "\n",
    "PROJECT_NAME = \"bccd_detection\"\n",
    "MODEL_NAME = PROJECT_NAME\n",
    "\n",
    "api = Api()\n",
    "proj = api.projects.create(\n",
    "    name=PROJECT_NAME,\n",
    "    description=\"Blood cell detection tutorial using YOLOX\",\n",
    "    exists_ok=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4e5f6a7",
   "metadata": {},
   "source": [
    "## 1. Load the Dataset\n",
    "\n",
    "`validate_model` pulls sample resources directly from a `DatamintBaseDataset`.\n",
    "No extra setup is needed — just pass `dataset=` and it handles the rest."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e5f6a7b8",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = build_dataset(project=PROJECT_NAME, allow_external_annotations=True)\n",
    "\n",
    "print(f\"Resources : {len(dataset.resources)}\")\n",
    "print(f\"Box labels: {dataset.box_labels_set}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6a7b8c9",
   "metadata": {},
   "source": [
    "## 2. Load the Registered Model\n",
    "\n",
    "Load any registered model from MLflow using its URI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "a7b8c9d0",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Downloading artifacts: 100%|██████████| 6/6 [00:27<00:00,  4.66s/it]   "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model      : YOLOXModule\n",
      "task_type  : TaskType.OBJECT_DETECTION\n",
      "annotation_specs: [AnnotationSpec(type=<AnnotationType.SQUARE: 'square'>, scope='image', required=False, identifier='Platelets'), AnnotationSpec(type=<AnnotationType.SQUARE: 'square'>, scope='image', required=False, identifier='RBC'), AnnotationSpec(type=<AnnotationType.SQUARE: 'square'>, scope='image', required=False, identifier='WBC')]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "model = datamint_flavor.load_model(f\"models:/{MODEL_NAME}/latest\")\n",
    "\n",
    "print(f\"Model      : {model.__class__.__name__}\")\n",
    "print(f\"task_type  : {model.task_type}\")\n",
    "print(f\"annotation_specs: {model.annotation_specs}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8c9d0e1",
   "metadata": {},
   "source": "## 3. Run Validation\n\n`validate_model` accepts either a dataset or a list of resources directly.\n\n- `dataset` — pulls `n_samples` resources automatically\n- `sample_input` — pass a `list[Resource]` directly"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c9d0e1f2",
   "metadata": {},
   "outputs": [],
   "source": [
    "report = validate_model(model, dataset=dataset, n_samples=2)\n",
    "print(report)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0e1f2a3",
   "metadata": {},
   "source": [
    "## 4. Reading the Report\n",
    "\n",
    "Each line maps to one check:\n",
    "\n",
    "| Marker | Meaning |\n",
    "|--------|---------|\n",
    "| `[v]` | Check passed, ready to deploy. |\n",
    "| `[!]` | Warning — model still deployable, but worth investigating |\n",
    "| `[x]` | Error — model should not be deployed as-is |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3b4c5d6",
   "metadata": {},
   "source": [
    "## 5. Passing Resources Directly\n",
    "\n",
    "If you already have specific resources you want to test against — for example,\n",
    "a known edge case — pass them via `sample_input` instead of a dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b4c5d6e7",
   "metadata": {},
   "outputs": [],
   "source": [
    "samples = list(dataset.resources[:3])\n",
    "\n",
    "report = validate_model(model, sample_input=samples)\n",
    "print(report)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "env",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}