{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Datamint Data Upload Tutorial\n",
    "\n",
    "This comprehensive tutorial covers all aspects of uploading data to Datamint, from single files to complex batch operations with metadata and segmentations.\n",
    "\n",
    "## 📋 Table of Contents\n",
    "\n",
    "| Section | Description | Key Features |\n",
    "|---------|-------------|--------------|\n",
    "| **[Setup & Connection](#setup--connection)** | Initialize API connection | Authentication, configuration |\n",
    "| **[1. Single Resource Upload](#1-single-resource-upload)** | Basic file upload with metadata | Tags, channels, anonymization |\n",
    "| **[2. Batch Upload with Different File Types](#2-batch-upload-with-different-file-types)** | Multiple files in one operation | Mixed formats, error handling |\n",
    "| **[Data Organization Guide](#data-organization)** | Best practices for organizing data | Tags, channels, projects |\n",
    "| **[3. Upload Segmentation](#3-upload-segmentation)** | Adding segmentation masks | Multi-class support, resource linking |\n",
    "| **[4. Upload Resource with Segmentations](#4-upload-resource-with-segmentations-in-one-request)** | Combined upload workflow | Efficiency, automatic linking |\n",
    "| **[5. Upload with JSON Metadata](#5-upload-with-json-metadata)** | Structured metadata inclusion | Custom fields, DICOM metadata |\n",
    "| **[6. Project Management](#6-project-management)** | Team collaboration features | Project creation, resource organization |\n",
    "| **[7. Downloading and Accessing Data](#7-downloading-and-accessing-data)** | Retrieve uploaded resources | Format conversion, annotations |\n",
    "| **[Next Steps](#next-steps)** | Additional resources and tutorials | Documentation, advanced features |\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup & Connection\n",
    "\n",
    "Initialize the Datamint API connection using the new **modular API client**. This provides a cleaner, more organized interface with dedicated modules for different operations:\n",
    "\n",
    "- `api.resources` - Resource management (upload, download, etc.)\n",
    "- `api.projects` - Project management and collaboration\n",
    "- `api.annotations` - Annotation handling and segmentations\n",
    "- `api.users` - User management\n",
    "\n",
    "Make sure you've run `datamint config` in your terminal first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "from datamint import Api\n",
    "\n",
    "# Creates a connection with the server.\n",
    "# Don't forget to run `datamint config` in a terminal, if you haven't already.\n",
    "# Or use api_key parameter in Api()\n",
    "api = Api()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Single Resource Upload\n",
    "\n",
    "Upload a single file with basic metadata and organization options."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Single file upload with comprehensive options\n",
    "dicom_file = '../data/Case14.dcm'\n",
    "new_resource_id = api.resources.upload_resource(\n",
    "    dicom_file,\n",
    "    channel='tutorial_channel',  # arbitrary channel name for organization\n",
    "    tags=['tutorial', 'case14'],  # tags for easy searching later\n",
    "    publish=False,  # set to True to bypass inbox and directly publish\n",
    "    anonymize=True,  # anonymize DICOM data (default for DICOM files)\n",
    ")\n",
    "\n",
    "print(f\"Uploaded resource ID: {new_resource_id}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### API Method Comparison\n",
    "\n",
    "| Method | Use Case |  Example |\n",
    "|--------|----------|---------|\n",
    "| `upload_resource()` | Single file upload | `api.resources.upload_resource('file.dcm')` |\n",
    "| `upload_resources()` | Batch upload | `api.resources.upload_resources(['file1.dcm', 'file2.png'])` |\n",
    "\n",
    "Both methods support the same parameters (tags, channels, segmentations, metadata, etc.)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get all the resources with specific tags\n",
    "all_resources = list(api.resources.get_list(\n",
    "    status='inbox',\n",
    "    tags=['tutorial']\n",
    "))\n",
    "\n",
    "print(f\"Found {len(all_resources)} resources with 'tutorial' tag\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Batch Upload with Different File Types\n",
    "\n",
    "Upload multiple files of different types in a single operation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Organization\n",
    "\n",
    "**Channels and Tags** are powerful tools for organizing your data:\n",
    "\n",
    "#### 🏷️ **Tags**\n",
    "- **Purpose**: Searchable labels for filtering and categorization\n",
    "- **Examples**: `['mri', 'brain', 'segmented']`, `['study_2024', 'patient_cohort_a']`\n",
    "- **Best Practice**: Use consistent naming conventions\n",
    "\n",
    "#### 📁 **Channels** \n",
    "- **Purpose**: Logical groupings for related resources\n",
    "- **Examples**: `'cardiac_study'`, `'preprocessing_pipeline'`, `'quality_control'`\n",
    "- **Best Practice**: One channel per study or workflow\n",
    "\n",
    "#### 🎯 **Projects**\n",
    "- **Purpose**: Collaborative workspaces with access control\n",
    "- **Features**: Resource collections, annotation workflows, team management\n",
    "- **Best Practice**: Create projects for specific research objectives"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Upload multiple files at once\n",
    "files_to_upload = [\n",
    "    '../data/Case14.dcm',\n",
    "    '../data/sample_image.png',  # Replace with actual image file\n",
    "    '../data/sample_video.mp4'   # Replace with actual video file\n",
    "]\n",
    "\n",
    "resource_ids = api.resources.upload_resources(\n",
    "    files_to_upload,\n",
    "    channel='batch_upload_demo',\n",
    "    tags=['batch', 'mixed_types'],\n",
    "    on_error='skip',  # Skip files that fail to upload\n",
    "    mung_filename='all'  # Include full path in filename\n",
    ")\n",
    "\n",
    "print(f\"Uploaded {len([r for r in resource_ids if not isinstance(r, Exception)])} files successfully\")\n",
    "for file, result in zip(files_to_upload, resource_ids):\n",
    "    if isinstance(result, Exception):\n",
    "        print(f\"Failed to upload {file}: {result}\")\n",
    "    else:\n",
    "        print(f\"✓ {Path(file).name} -> {result}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Upload Segmentation\n",
    "\n",
    "**Objective**: Add segmentation masks to existing resources for machine learning and analysis.\n",
    "\n",
    "### Segmentation Features:\n",
    "- 🎨 **Multi-class Support**: Handle multiple anatomical regions\n",
    "- 🔗 **Resource Linking**: Associate segmentations with source images  \n",
    "- 📏 **Format Flexibility**: Support for NIfTI, PNG, and numpy arrays\n",
    "- 🏷️ **Class Naming**: Map pixel values to meaningful labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Upload segmentation with comprehensive class mapping\n",
    "seg_file = '../data/Case14_Bones.nii.gz'\n",
    "resource_id = new_resource_id\n",
    "\n",
    "# Define pixel value to anatomical region mapping\n",
    "class_names = {\n",
    "    1: \"Femur\",      # Pixel value 1 represents femur\n",
    "    2: \"Tibia\",       # Pixel value 2 represents tibia\n",
    "}\n",
    "\n",
    "# Use the new annotations API with the upload_segmentations method\n",
    "segmentation_ids = api.annotations.upload_segmentations(\n",
    "    resource=resource_id,\n",
    "    file_path=seg_file,\n",
    "    name=class_names,\n",
    "    imported_from='manual_annotation',  # Track the source of annotations\n",
    "    # model_id='MyModelName' # Optional. Associate them to a model. See ``mlflow.search_registered_models()`` or ``api.models.get_all()``\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Upload Resource with Segmentations in One Request\n",
    "\n",
    "**Objective**: Optimize workflow by uploading resources and segmentations simultaneously.\n",
    "\n",
    "### Advantages of Combined Upload:\n",
    "- ⚡ **Efficiency**: Single API call for related data\n",
    "- 🔗 **Automatic Linking**: Resources and segmentations are pre-associated\n",
    "- 🛡️ **Atomicity**: Either both succeed or both fail\n",
    "- 📊 **Progress Tracking**: Unified upload monitoring"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dicom_file = '../data/Case14.dcm'\n",
    "class_names = {\n",
    "    1: \"Femur\",\n",
    "    2: \"Tibia\"\n",
    "}\n",
    "\n",
    "# These are the segmentation files that will be uploaded to the server\n",
    "segfiles = {\n",
    "    'files': ['../data/Case14_Bones.nii.gz'], # same number of frames as the image file\n",
    "    'names': class_names  # mapping pixel values to class names\n",
    "}\n",
    "\n",
    "new_resource_id = api.resources.upload_resource(\n",
    "    dicom_file,\n",
    "    segmentation_files=segfiles,\n",
    "    channel='with_segmentation',\n",
    "    tags=['tutorial', 'with_seg'],\n",
    "    publish=False\n",
    ")\n",
    "\n",
    "print(f\"Uploaded resource with segmentation: {new_resource_id}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Upload with JSON Metadata\n",
    "\n",
    "Include structured metadata with your uploads. This is particularly useful for research data with custom fields.\n",
    "\n",
    "* **Supported Metadata Fields:** Common DICOM and research fields are automatically recognized and indexed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nifti_file = '../data/my_niftidata.nii.gz'\n",
    "\n",
    "# Create and upload resources with structured JSON metadata\n",
    "metadata_example = {\n",
    "    # Core identifiers\n",
    "    \"SeriesInstanceUID\": \"1.2.3.4.5.6.7.8.9.10.TUTORIAL.001\",\n",
    "    \"StudyInstanceUID\": \"1.2.3.4.5.6.7.8.9.STUDY.001\",\n",
    "\n",
    "    # Clinical information\n",
    "    \"patient_age\": 45,\n",
    "    \"acquisition_date\": \"2024-01-15\",\n",
    "    \"scanner_model\": \"Example Scanner 3T\",\n",
    "    \"modality\": \"CT\"\n",
    "}\n",
    "# Upload with metadata\n",
    "resource_with_metadata = api.resources.upload_resource(\n",
    "    nifti_file,\n",
    "    channel='with_metadata',\n",
    "    tags=['tutorial', 'metadata_example'],\n",
    "    metadata=metadata_example\n",
    ")\n",
    "\n",
    "print(f\"Uploaded resource with metadata: {resource_with_metadata}\")\n",
    "\n",
    "# Verify the metadata was included\n",
    "resource_info = api.resources.get_by_id(resource_with_metadata)\n",
    "print(\"Resource modality:\", resource_info.modality)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Project Management\n",
    "\n",
    "**Objective**: Organize resources into collaborative projects for team-based workflows.\n",
    "\n",
    "### Project Features:\n",
    "- 👥 **Team Collaboration**: Shared access and work\n",
    "- 📁 **Resource Organization**: Logical grouping of related data\n",
    "- 🔄 **Workflow Management**: Annotation tasks and review processes\n",
    "- 📊 **Progress Tracking**: Monitor project completion status"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get some resources to add to a project\n",
    "tutorial_resources = list(api.resources.get_list(\n",
    "    tags=['tutorial'],\n",
    "    status='inbox'\n",
    "))\n",
    "\n",
    "if tutorial_resources:\n",
    "    resource_ids_for_project = [r.id for r in tutorial_resources[:3]]  # Take first 3 resources\n",
    "\n",
    "    # Create a new project (exists_ok returns the existing project instead of raising)\n",
    "    project = api.projects.create(\n",
    "        name=\"Tutorial Project\",\n",
    "        description=\"A project created for demonstration purposes\",\n",
    "        resource_ids=resource_ids_for_project,\n",
    "        exists_ok=True\n",
    "    )\n",
    "\n",
    "    print(f\"Created project: {project.name} (ID: {project.id})\")\n",
    "\n",
    "    # List all projects\n",
    "    all_projects = api.projects.get_list()\n",
    "    print(f\"\\nAll projects ({len(all_projects)}):\")\n",
    "    for proj in all_projects:\n",
    "        print(f\"  - {proj.name} (ID: {proj.id})\")\n",
    "else:\n",
    "    print(\"No tutorial resources found to add to project\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. Downloading and Accessing Data\n",
    "\n",
    "**Objective**: Retrieve and work with uploaded resources, including format conversion and annotation access.\n",
    "\n",
    "### Download Features:\n",
    "- 📥 **Format Flexibility**: Raw bytes, auto-converted objects, or saved files\n",
    "- 🔄 **Type Conversion**: Automatic conversion to appropriate data types\n",
    "- 📊 **Metadata Access**: Retrieve associated annotations and metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Download a resource file\n",
    "api.resources.download_resource_file(\n",
    "    new_resource_id,\n",
    "    auto_convert=False,\n",
    "    save_path='downloaded_resource.dcm'  # Save to a specific file\n",
    ")\n",
    "\n",
    "# Download and auto-convert (for DICOM files, returns pydicom Dataset)\n",
    "resource_object = api.resources.download_resource_file(\n",
    "    new_resource_id,\n",
    "    auto_convert=True\n",
    ")\n",
    "print(f\"Auto-converted to: {type(resource_object)}\")  # `pydicom.Dataset` object\n",
    "\n",
    "# Get annotations for this resource\n",
    "annotations = list(api.annotations.get_list(resource=new_resource_id))\n",
    "for ann in annotations:\n",
    "    print(f\"  - {ann.identifier}: {ann.type}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tutorial covered the main features of the Datamint Python API. For more advanced usage:\n",
    "\n",
    "**Check the full documentation**: https://sonanceai.github.io/datamint-python-api/\n",
    "\n",
    "Happy coding! 🚀"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}