{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Importing External Dataset Formats\n",
    "\n",
    "This tutorial shows how to bring an already-labeled dataset into a Datamint project using `datamint.importers`, instead of hand-rolling `upload_resources()` plus a loop of `add_box_annotation()` calls.\n",
    "\n",
    "Three formats are supported:\n",
    "\n",
    "| Importer | Format |\n",
    "|---|---|\n",
    "| `COCOImporter` | COCO JSON (`images`/`annotations`/`categories`) |\n",
    "| `PascalVOCImporter` | Pascal VOC XML (one `.xml` file per image) |\n",
    "| `YOLOImporter` | YOLO `.txt` labels (normalized `class x_center y_center width height`) |\n",
    "\n",
    "All three share the same two-step shape:\n",
    "- `.parse()` reads and validates the dataset on disk. Use it to preview image/box counts and class names before uploading anything. Unsupported annotations (e.g. polygons) are counted, not silently dropped.\n",
    "- `.import_to_project(project)` reuses the parsed result to upload the images and their box annotations. \n",
    "\n",
    "This notebook builds tiny synthetic datasets in each format (a couple of generated images) so it runs end-to-end without needing an external download."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "Make sure you've run `datamint config` in a terminal (or set the `DATAMINT_API_KEY` environment variable) before running this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tempfile\n",
    "from pathlib import Path\n",
    "\n",
    "from datamint import Api\n",
    "\n",
    "api = Api()\n",
    "\n",
    "workdir = Path(tempfile.mkdtemp(prefix=\"datamint_import_tutorial_\"))\n",
    "print(f\"Working directory: {workdir}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The three sections below each build their own tiny 2-image dataset from this shared layout, so they can be run independently."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from PIL import Image, ImageDraw\n",
    "\n",
    "IMAGE_SIZE = (128, 128)\n",
    "\n",
    "# file_name -> (label, (x, y, width, height))\n",
    "DEMO_BOXES = {\n",
    "    \"image_0.png\": (\"cat\", (10, 10, 40, 30)),\n",
    "    \"image_1.png\": (\"dog\", (20, 15, 35, 25)),\n",
    "}\n",
    "\n",
    "\n",
    "def new_demo_image(bbox):\n",
    "    img = Image.new(\"RGB\", IMAGE_SIZE, color=\"white\")\n",
    "    x, y, w, h = bbox\n",
    "    ImageDraw.Draw(img).rectangle([x, y, x + w, y + h], outline=\"black\")\n",
    "    return img"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll import each format into its own project, so the results are easy to tell apart in the web app."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "project_coco = api.projects.create(\n",
    "    \"Import Tutorial - COCO\", description=\"datamint.importers tutorial\", exists_ok=True\n",
    ")\n",
    "project_voc = api.projects.create(\n",
    "    \"Import Tutorial - Pascal VOC\", description=\"datamint.importers tutorial\", exists_ok=True\n",
    ")\n",
    "project_yolo = api.projects.create(\n",
    "    \"Import Tutorial - YOLO\", description=\"datamint.importers tutorial\", exists_ok=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Import a COCO dataset\n",
    "\n",
    "`COCOImporter(annotations_file, images_dir=None)` reads a single COCO JSON file. `images_dir` defaults to the annotations file's parent directory.\n",
    "\n",
    "First, generate a tiny COCO dataset: two images, one box each.\n",
    "\n",
    "*(`build_coco_dataset` below is demo-data setup only, not part of the importer API -- feel free to skip to the `COCOImporter` cell.)*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "\n",
    "def build_coco_dataset(workdir: Path) -> Path:\n",
    "    coco_dir = workdir / \"coco_dataset\"\n",
    "    coco_dir.mkdir()\n",
    "\n",
    "    images, annotations = [], []\n",
    "    for i, (file_name, (label, bbox)) in enumerate(DEMO_BOXES.items()):\n",
    "        new_demo_image(bbox).save(coco_dir / file_name)\n",
    "\n",
    "        images.append({\"id\": i, \"file_name\": file_name, \"width\": IMAGE_SIZE[0], \"height\": IMAGE_SIZE[1]})\n",
    "        annotations.append({\n",
    "            \"id\": i,\n",
    "            \"image_id\": i,\n",
    "            \"category_id\": 0 if label == \"cat\" else 1,\n",
    "            \"bbox\": list(bbox),\n",
    "        })\n",
    "\n",
    "    coco_json = {\n",
    "        \"images\": images,\n",
    "        \"annotations\": annotations,\n",
    "        \"categories\": [{\"id\": 0, \"name\": \"cat\"}, {\"id\": 1, \"name\": \"dog\"}],\n",
    "    }\n",
    "    annotations_file = coco_dir / \"_annotations.coco.json\"\n",
    "    annotations_file.write_text(json.dumps(coco_json))\n",
    "    return annotations_file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "coco_annotations_file = build_coco_dataset(workdir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Parse it first to preview what would be uploaded:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datamint import COCOImporter\n",
    "\n",
    "coco_importer = COCOImporter(coco_annotations_file)\n",
    "\n",
    "preview = coco_importer.parse()\n",
    "print(f\"{preview.num_images} images, {preview.num_boxes} boxes, classes={preview.class_names}\")\n",
    "print(f\"missing images: {preview.missing_images}, unsupported annotations: {preview.unsupported_annotations}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then upload the images and their box annotations. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = coco_importer.import_to_project(project_coco, tags=[\"coco-import-tutorial\"])\n",
    "\n",
    "print(f\"Uploaded images: {result.n_images_uploaded}\")\n",
    "print(f\"Uploaded boxes: {result.n_boxes_uploaded}\")\n",
    "print(f\"Errors: {result.errors}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Import a Pascal VOC dataset\n",
    "\n",
    "`PascalVOCImporter(annotations_dir, images_dir)` -- unlike COCO, both directories are required explicitly.\n",
    "\n",
    "*(`build_voc_dataset` below is demo-data setup only, not part of the importer API -- feel free to skip to the `PascalVOCImporter` cell.)*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import xml.etree.ElementTree as ET\n",
    "\n",
    "\n",
    "def build_voc_dataset(workdir: Path) -> tuple[Path, Path]:\n",
    "    voc_dir = workdir / \"voc_dataset\"\n",
    "    images_dir = voc_dir / \"JPEGImages\"\n",
    "    annotations_dir = voc_dir / \"Annotations\"\n",
    "    images_dir.mkdir(parents=True)\n",
    "    annotations_dir.mkdir(parents=True)\n",
    "\n",
    "    for file_name, (label, bbox) in DEMO_BOXES.items():\n",
    "        new_demo_image(bbox).save(images_dir / file_name)\n",
    "\n",
    "        x, y, w, h = bbox\n",
    "        annotation = ET.Element(\"annotation\")\n",
    "        ET.SubElement(annotation, \"filename\").text = file_name\n",
    "        obj = ET.SubElement(annotation, \"object\")\n",
    "        ET.SubElement(obj, \"name\").text = label\n",
    "        bndbox = ET.SubElement(obj, \"bndbox\")\n",
    "        ET.SubElement(bndbox, \"xmin\").text = str(x)\n",
    "        ET.SubElement(bndbox, \"ymin\").text = str(y)\n",
    "        ET.SubElement(bndbox, \"xmax\").text = str(x + w)\n",
    "        ET.SubElement(bndbox, \"ymax\").text = str(y + h)\n",
    "        ET.ElementTree(annotation).write(annotations_dir / f\"{Path(file_name).stem}.xml\")\n",
    "\n",
    "    return annotations_dir, images_dir"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "voc_annotations_dir, voc_images_dir = build_voc_dataset(workdir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datamint import PascalVOCImporter\n",
    "\n",
    "voc_importer = PascalVOCImporter(voc_annotations_dir, voc_images_dir)\n",
    "\n",
    "preview = voc_importer.parse()\n",
    "print(f\"{preview.num_images} images, {preview.num_boxes} boxes, classes={preview.class_names}\")\n",
    "\n",
    "result = voc_importer.import_to_project(project_voc, tags=[\"voc-import-tutorial\"])\n",
    "print(f\"Uploaded images: {result.n_images_uploaded}, boxes: {result.n_boxes_uploaded}, errors: {result.errors}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Import a YOLO dataset\n",
    "\n",
    "`YOLOImporter(images_dir, labels_dir, class_names=None, data_yaml=None)` -- YOLO label files are pure numbers with no class name or image filename embedded, so class names must be resolved from one of: an explicit `class_names` list, an auto-detected `data.yaml`/`data.yml` (`names:` key), or an auto-detected legacy `classes.txt`. Coordinates are normalized (`0..1` of image width/height): `class x_center y_center width height`.\n",
    "\n",
    "*(`build_yolo_dataset` below is demo-data setup only, not part of the importer API -- feel free to skip to the `YOLOImporter` cell.)*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_yolo_dataset(workdir: Path) -> tuple[Path, Path, list[str]]:\n",
    "    yolo_dir = workdir / \"yolo_dataset\"\n",
    "    images_dir = yolo_dir / \"images\"\n",
    "    labels_dir = yolo_dir / \"labels\"\n",
    "    images_dir.mkdir(parents=True)\n",
    "    labels_dir.mkdir(parents=True)\n",
    "\n",
    "    class_names = [\"cat\", \"dog\"]\n",
    "    img_w, img_h = IMAGE_SIZE\n",
    "\n",
    "    for file_name, (label, bbox) in DEMO_BOXES.items():\n",
    "        new_demo_image(bbox).save(images_dir / file_name)\n",
    "\n",
    "        x, y, w, h = bbox\n",
    "        cx, cy = (x + w / 2) / img_w, (y + h / 2) / img_h\n",
    "        nw, nh = w / img_w, h / img_h\n",
    "\n",
    "        class_id = class_names.index(label)\n",
    "        label_path = labels_dir / f\"{Path(file_name).stem}.txt\"\n",
    "        label_path.write_text(f\"{class_id} {cx} {cy} {nw} {nh}\\n\")\n",
    "\n",
    "    return images_dir, labels_dir, class_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "yolo_images_dir, yolo_labels_dir, yolo_class_names = build_yolo_dataset(workdir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datamint import YOLOImporter\n",
    "\n",
    "yolo_importer = YOLOImporter(yolo_images_dir, yolo_labels_dir, class_names=yolo_class_names)\n",
    "\n",
    "preview = yolo_importer.parse()\n",
    "print(f\"{preview.num_images} images, {preview.num_boxes} boxes, classes={preview.class_names}\")\n",
    "\n",
    "result = yolo_importer.import_to_project(project_yolo, tags=[\"yolo-import-tutorial\"])\n",
    "print(f\"Uploaded images: {result.n_images_uploaded}, boxes: {result.n_boxes_uploaded}, errors: {result.errors}\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}