{ "cells": [ { "cell_type": "markdown", "id": "a1b2c3d4", "metadata": {}, "source": [ "# Patient-wise Dataset Splitting\n", "\n", "This notebook shows how to use `group_by_patient()` and `split(by_patient=True)` on a DICOM dataset.\n", "\n", "## The Problem: Data Leakage\n", "\n", "Standard random splitting shuffles individual resources. If a patient has **multiple scans** (e.g. a baseline CT and a follow-up CT), both scans can end up in different splits (one in `train`, one in `test`). Because the images share patient anatomy, the model effectively sees the test patient during training, inflating metrics.\n", "\n", "**Patient-wise splitting** fixes this by shuffling *patients* instead of *resources*: every scan belonging to a patient lands in the same split.\n", "\n", "```\n", "Resource-wise (risky) Patient-wise (safe)\n", "────────────────────── ──────────────────────\n", "train: scan_A1, scan_B1 train: patient_A (scan_A1, scan_A2)\n", "test: scan_A2, scan_B2 test: patient_B (scan_B1, scan_B2)\n", " ↑ same patient! ↑ clean boundary\n", "```\n", "\n", "## What You'll Learn\n", "\n", "1. How `patient_id` is exposed on resources\n", "2. `group_by_patient()` — organise a dataset by patient for inspection or custom splits\n", "3. `split(by_patient=True)` — an option for the split\n", "4. How to persist patient-wise splits to the server and reload them reproducibly\n", "5. How to handle resources without a `patient_id`" ] }, { "cell_type": "markdown", "id": "b2c3d4e5", "metadata": {}, "source": [ "## 1. Setup\n", "\n", "We use a small set of DICOM files bundled with **pydicom** as example data. Each file already carries a `PatientID` tag in its header.\n", "\n", "The upload runs only once: if the project already exists we skip straight to loading the dataset." ] }, { "cell_type": "code", "execution_count": 7, "id": "c3d4e5f6", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Project 'split_save_test' already exists, skipping upload.\n", "\n", "Loaded 20 resources\n" ] } ], "source": [ "from pathlib import Path\n", "\n", "import pydicom\n", "import pydicom.data\n", "\n", "from datamint import Api\n", "from datamint.dataset import ImageDataset\n", "\n", "PROJECT_NAME = \"split_save_test\"\n", "\n", "# 8 patients selected from pydicom's \n", "SELECTED_PATIENTS = {\n", " '4MR1', '8NM1', '13US1', 'CQ500-CT-310',\n", " '1CT1', '021234567', '11-05-25-142825', '204',\n", "}\n", "\n", "def _collect_dicom_files() -> list[str]:\n", " data_root = Path(pydicom.data.DATA_ROOT) / 'test_files'\n", " selected = []\n", " for f in sorted(data_root.glob('*.dcm')):\n", " try:\n", " ds = pydicom.dcmread(str(f), stop_before_pixels=True, force=True)\n", " if getattr(ds, 'PatientID', None) in SELECTED_PATIENTS:\n", " selected.append(str(f))\n", " except Exception:\n", " pass\n", " return selected\n", "\n", "api = Api()\n", "\n", "project = api.projects.get_by_name(PROJECT_NAME)\n", "if project is None:\n", " project = api.projects.create(PROJECT_NAME, description=\"Patient Split Demo\")\n", " dicom_files = _collect_dicom_files()\n", " print(f\"Uploading {len(dicom_files)} DICOM files for {len(SELECTED_PATIENTS)} patients...\")\n", " api.resources.upload_resources(\n", " dicom_files,\n", " publish_to=project,\n", " assemble_dicoms=False,\n", " progress_bar=True,\n", " )\n", " print(\"Upload complete.\")\n", "else:\n", " print(f\"Project '{PROJECT_NAME}' already exists, skipping upload.\")\n", "\n", "dataset = ImageDataset(project=project, include_unannotated=True)\n", "print(f\"\\nLoaded {len(dataset)} resources\")" ] }, { "cell_type": "markdown", "id": "d4e5f6a7", "metadata": {}, "source": [ "## 2. Inspecting `patient_id`\n", "\n", "For DICOM files the platform extracts `PatientID` from the DICOM header automatically at upload time and stores it as a top-level field on the resource. No manual assignment needed." ] }, { "cell_type": "code", "execution_count": 8, "id": "e5f6a7b8", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "MR_truncated.dcm patient_id='4MR1'\n", "MR_small_RLE.dcm patient_id='4MR1'\n", "JPEG2000-embedded-sequence-delimiter.dcm patient_id='8NM1'\n", "JPEG2000.dcm patient_id='8NM1'\n", "MR_small.dcm patient_id='4MR1'\n" ] } ], "source": [ "for r in dataset.resources[:5]:\n", " print(f\"{r.filename:40s} patient_id={r.patient_id!r}\")" ] }, { "cell_type": "markdown", "id": "a7b8c9d0", "metadata": {}, "source": [ "## 3. `group_by_patient()`\n", "\n", "`group_by_patient()` returns a `dict` mapping each patient ID to a sub-dataset containing only that patient's resources.\n", "\n", "### Handling `patient_id=None`\n", "\n", "Well-formed DICOM files always carry a `PatientID` tag, so all resources here should have one. In edge cases, anonymised DICOMs where the tag was stripped, `patient_id` will be `None`. The `none_patient_id_strategy` parameter controls what to do:\n", "\n", "| Strategy | Behaviour |\n", "|---|---|\n", "| `'individual'` (default) | Each `None`-patient resource becomes its own group |\n", "| `'group'` | All grouped together under key `None` |\n", "| `'skip'` | Excluded from the output entirely |\n", "| `'error'` | Raises `ValueError` immediately |" ] }, { "cell_type": "code", "execution_count": 9, "id": "b8c9d0e1", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Patient groups : 8\n", "Total resources: 20\n", "Per-group size : min=1, max=9\n", "\n", " 4MR1 9 resource(s)\n", " 8NM1 4 resource(s)\n", " 1CT1 1 resource(s)\n", " 13US1 2 resource(s)\n", " CQ500-CT-310 1 resource(s)\n", " 11-05-25-142825 1 resource(s)\n", " 021234567 1 resource(s)\n", " 204 1 resource(s)\n" ] } ], "source": [ "groups = dataset.group_by_patient()\n", "\n", "sizes = [len(g) for g in groups.values()]\n", "print(f\"Patient groups : {len(groups)}\")\n", "print(f\"Total resources: {sum(sizes)}\")\n", "print(f\"Per-group size : min={min(sizes)}, max={max(sizes)}\")\n", "print()\n", "for pid, g in groups.items():\n", " print(f\" {pid:20s} {len(g):2d} resource(s)\")" ] }, { "cell_type": "markdown", "id": "c9d0e1f2", "metadata": {}, "source": [ "## 4. `split(by_patient=True)` — Compute Patient-wise Splits Locally\n", "\n", "When passing by_patient=True, the split shuffles patients instead of individual resources. Internally it shuffles **patients**, assigns patient buckets to splits by ratio, then collects all resources from each bucket." ] }, { "cell_type": "code", "execution_count": 10, "id": "d0e1f2a3", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "train : 7 resources\n", "val : 9 resources\n", "test : 4 resources\n" ] } ], "source": [ "parts = dataset.split(\n", " train=0.7,\n", " val=0.15,\n", " test=0.15,\n", " by_patient=True,\n", " seed=42,\n", ")\n", "\n", "for name, ds in parts.items():\n", " print(f\"{name:6s}: {len(ds):4d} resources\")" ] }, { "cell_type": "markdown", "id": "f2a3b4c5", "metadata": {}, "source": [ "## 5. Persist Splits to the Server\n", "\n", "The `parts` object returned by `split()` is a `SplitResult` — it behaves exactly like a dict, but it also knows how to save itself to the server via `.save()`.\n", "\n", "Once persisted, anyone can reload exactly the same split (without re-running the patient split logic) by calling `dataset.split()` with no arguments.\n", "\n", "If the project already has split assignments, `.save()` raises an error. Pass `force=True` to overwrite." ] }, { "cell_type": "code", "execution_count": 11, "id": "a3b4c5d6", "metadata": {}, "outputs": [], "source": [ "parts.save()\n", "\n", "# If the project already has splits, this raises:\n", "# ValueError: Project '...' already has split assignments. Use save(force=True) to overwrite.\n", "# To overwrite:\n", "# parts.save(force=True)" ] }, { "cell_type": "markdown", "id": "b4c5d6e7", "metadata": {}, "source": [ "## 6. Reload Splits from the Server\n", "\n", "After persisting, the splits are loaded back just like any project-scoped split, no `by_patient`, no ratios. The `split_as_of_timestamp` returned on each sub-dataset can be stored and replayed later to recover the exact same snapshot." ] }, { "cell_type": "code", "execution_count": 12, "id": "c5d6e7f8", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "val : 9 resources\n", "test : 4 resources\n", "train : 7 resources\n" ] } ], "source": [ "# Reload the dataset fresh (simulates a new session)\n", "dataset_fresh = ImageDataset(project=project, include_unannotated=True)\n", "\n", "# No by_patient, no ratios, reads from project API\n", "reloaded = dataset_fresh.split()\n", "\n", "for name, ds in reloaded.items():\n", " print(f\"{name:6s}: {len(ds):4d} resources\")" ] }, { "cell_type": "code", "execution_count": 13, "id": "d6e7f8a9", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Replaying split snapshot from 2026-06-19T16:37:40.622014Z\n", "{'val': 9, 'test': 4, 'train': 7}\n" ] } ], "source": [ "# Replay the exact same snapshot at any later point\n", "snapshot_ts = reloaded['train'].split_as_of_timestamp\n", "\n", "replayed = dataset_fresh.split(as_of_timestamp=snapshot_ts)\n", "print(f\"Replaying split snapshot from {snapshot_ts}\")\n", "print({name: len(ds) for name, ds in replayed.items()})" ] } ], "metadata": { "kernelspec": { "display_name": "env", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.3" } }, "nbformat": 4, "nbformat_minor": 5 }