datamint.Experiment

class datamint.experiment.experiment.Experiment(name, project_name=None, description=None, api_key=None, root_url=None, dataset_dir=None, log_enviroment=True, dry_run=False, auto_log=True, tags=None, allow_existing=False)

Experiment class to log metrics, models, and other information to the platform.

Parameters:

name (str) – Name of the experiment.
project_name (str) – Name of the project.
description (str) – Description of the experiment.
api_key (str) – API key for the platform.
root_url (str) – Root URL of the platform.
dataset_dir (str) – Directory to store the datasets.
log_enviroment (bool) – Log the enviroment information.
dry_run (bool) – Run in dry-run mode. No data will be uploaded to the platform
auto_log (bool) – Automatically log the experiment using patching mechanisms.
tags (List[str]) – Tags to add to the experiment.
allow_existing (bool)

finish(): Finish the experiment. This will log the summary and finish the experiment.

get_dataset(split='all', **kwargs)

Get the dataset associated with the experiment’s project. The dataset will be downloaded to the directory specified in the constructor (self.dataset_dir).

Parameters:

split (str) – The split of the dataset to get. Can be one of [‘all’, ‘train’, ‘test’, ‘val’].
**kwargs – Additional arguments to pass to the DatamintDataset class.

Returns:

The dataset object.

Return type:

DatamintDataset

static get_enviroment_info()

Get the enviroment information of the machine such as OS, Python version, etc.

Returns:: Enviroment information.
Return type:: Dict

log_classification_predictions(predictions_conf, resource_ids, label_names, dataset_split=None, frame_idxs=None, step=None, epoch=None, add_info=None)

Log the classification predictions of the model.

Parameters:

predictions_conf (np.ndarray) –
The predictions of the model. Can have two shapes:
- Shape (N, C) where N is the number of samples and C is the number of classes. Does not need to sum to 1 (i.e., can be multilabel).
- Shape (N,) where N is the number of samples. In this case, label_names should have the same length as the predictions.
label_names (List[str]) – The names of the classes. If the predictions are shape (N,), this should have the same length as the predictions.
resource_ids (List[str]) – The resource IDs of the samples.
dataset_split (Optional[str]) – The dataset split of the predictions.
frame_idxs (Optional[List[int]]) – The frame indexes of the predictions.
step (Optional[int]) – The step of the experiment.
epoch (Optional[int]) – The epoch of the experiment.
add_info (Optional[Dict]) – Additional information to add to each prediction.

Example

predictions_conf = np.array([[0.9, 0.1], [0.2, 0.8]])
label_names = ['cat', 'dog']
resource_ids = ['123', '456']
exp.log_classification_predictions(predictions_conf, label_names, resource_ids, dataset_split='test')

log_dataset_stats(dataset, dataset_entry_name='default')

Log the statistics of the dataset.

Parameters:

dataset (DatamintDataset) – The dataset to log the statistics.
dataset_entry_name (str) – The name of the dataset entry. Used to distinguish between different datasets and dataset splits.

Example

dataset = exp.get_dataset(split='train')
exp.log_dataset_stats(dataset, dataset_entry_name='train')

log_metric(name, value, step=None, epoch=None, show_in_summary=False)

Log a metric to the platform.

Parameters:

name (str) – Arbritary name of the metric.
value (float) – Value of the metric.
step (int) – The step of the experiment.
epoch (int) – The epoch of the experiment.
show_in_summary (bool) – Show the metric in the summary. Use this to show only important metrics in the summary.

Return type:

None

Example

>>> exp.log_metric('test/sensitivity', 0.9, show_in_summary=True)

See also

log_metric()

log_model(model, hyper_params=None, log_model_attributes=True, torch_save_kwargs={})

Log the model to the platform.

Parameters:

model (torch.nn.Module | str | IO[bytes]) – The model to log. Can be a torch model, a path to a .pt or .pth file, or a BytesIO object.
hyper_params (Optional[Dict]) – The hyper-parameters of the model. Arbitrary key-value pairs.
log_model_attributes (bool) – Adds the attributes of the model to the hyper-parameters.
torch_save_kwargs (Dict) – Additional arguments to pass to torch.save.

Example

exp.log_model(model, hyper_params={"num_layers": 3, "pretrained": True})

log_segmentation_predictions(resource_id, predictions, label_name, frame_index=None, threshold=0.5, predictions_format='probability')

Log the segmentation prediction of the model for a single frame

Parameters:

resource_id (str | dict) – The resource ID of the sample.
predictions (ndarray | str) – The predictions of the model. One binary mask for each class. Can be a numpy array of shape (H, W) or (N,H,W); Or a path to a png file; Or a path to a .nii.gz file.
label_name (str | dict[int, str]) – The name of the class or a dictionary mapping pixel values to names. Example: {1: 'Femur', 2: 'Tibia'} means that pixel value 1 is ‘Femur’ and pixel value 2 is ‘Tibia’.
frame_index (int | list[int] | None) – The frame index of the prediction or a list of frame indexes. If a list, must have the same length as the predictions. If None,
threshold (float) – The threshold to apply to the predictions.
predictions_format (Literal['multi-class', 'probability']) – The format of the predictions. Can be a probability mask (‘probability’) or a multi-class mask (‘multi-class’).

Example

resource_id = '123'
predictions = np.array([[0.1, 0.4], [0.9, 0.2]])
label_name = 'fracture'
exp.log_segmentation_predictions(resource_id, predictions, label_name, threshold=0.5)

resource_id = '456'
predictions = np.array([[0, 1, 2], [1, 2, 0]])  # Multi-class mask with values 0, 1, 2
label_name = {1: 'Femur', 2: 'Tibia'}  # Mapping of pixel values to class names
exp.log_segmentation_predictions(
    resource_id,
    predictions,
    label_name,
    predictions_format='multi-class'
)

log_semantic_seg_predictions(predictions, resource_ids, label_names, dataset_split=None, frame_idxs=None, step=None, epoch=None, threshold=0.5)

Log the semantic segmentation predictions of the model.

Parameters:

predictions (np.ndarray | str) – The predictions of the model. A list of numpy arrays of shape (N, C, H, W). Or a path to a png file; Or a path to a .nii.gz file.
label_names (list[str]) – The names of the classes. List of strings of size C.
resource_ids (list[str]) – The resource IDs of the samples.
dataset_split (Optional[str]) – The dataset split of the predictions.
frame_idxs (Optional[list[int]]) – The frame indexes of the predictions.
step (Optional[int]) – The step of the experiment.
epoch (Optional[int]) – The epoch of the experiment.
threshold (float)

log_summary(result_summary)

Log the summary of the experiment. This is what will be shown in the platform summary.

Parameters:: result_summary (Dict) – The summary of the experiment.
Return type:: None

Example

exp.log_summary({"metrics": {
                    "test/F1Score": 0.85,
                    "test/Accuracy": 0.9,
                    "test/Sensitivity": 0.92,
                    "test/Positive Predictive Value": 0.79,
                    }
                })

set_model(model, hyper_params=None)

Set the model and hyper-parameters of the experiment.

Parameters:

model (torch.nn.Module) – The model to log.
hyper_params (Optional[Dict]) – The hyper-parameters of the model.

update_summary_metrics(phase, f1score, accuracy, sensitivity, ppv)

Handy method to update the summary with the most common classification metrics.

Parameters:

phase (str) – The phase of the experiment. Can be ‘train’, ‘val’, ‘test’, ‘’, or None.
f1score (float) – The F1 score.
accuracy (float) – The accuracy.
sensitivity (float) – The sensitivity (a.k.a recall).
specificity (float) – The specificity.
ppv (float) – The positive predictive value (a.k.a precision).