{
"cells": [
{
"cell_type": "markdown",
"id": "metrics-introduction",
"metadata": {},
"source": [
"# Evaluate segmentation results\n",
"\n",
"Compare saved PanopTILs predictions with the matching validation labels. Measure\n",
"how well nuclei are detected, how closely their masks overlap, and which cell\n",
"types need improvement. Evaluation runs on CPU; it does not rerun the model.\n",
"\n",
"The example model was trained for only five epochs and is still undertrained.\n",
"Its low scores, especially for some cell types, demonstrate how to compute and\n",
"interpret the metrics. They are not a benchmark of fully trained models.\n",
"\n",
"Use the masks saved by [image inference](inference.ipynb). Keep one `.npz` file\n",
"per image in `predictions/`, named after the source image, with `instances` and\n",
"`classes` arrays. The example outputs use 16 held-out validation regions.\n"
]
},
{
"cell_type": "markdown",
"id": "metrics-setup-heading",
"metadata": {},
"source": [
"## Setup\n"
]
},
{
"cell_type": "code",
"id": "metrics-setup",
"metadata": {},
"source": [
"from pathlib import Path\n",
"\n",
"import cv2\n",
"import matplotlib.pyplot as plt\n",
"import numpy as np\n",
"import polars as pl\n",
"from matplotlib.patches import Patch\n",
"from skimage.color import label2rgb\n",
"from skimage.segmentation import find_boundaries\n",
"\n",
"from cellseg_models_pytorch.metrics import (\n",
" dice_coef,\n",
" get_stats,\n",
" iou_score,\n",
" panoptic_quality,\n",
")\n",
"from cellseg_models_pytorch.utils import FileHandler\n",
"\n",
"data_root = Path(\"../../data/panoptils-nuclei\")\n",
"prediction_root = Path(\"predictions\")\n",
"example_images = [\"region_1252\", \"region_0804\"]\n",
"class_names = [\n",
" \"untyped\",\n",
" \"neoplastic\",\n",
" \"stromal\",\n",
" \"inflammatory\",\n",
" \"epithelial\",\n",
" \"other\",\n",
" \"unknown\",\n",
"]\n",
"\n",
"# Use the same class colours in reference and prediction overlays.\n",
"class_palette = np.array(\n",
" [\n",
" [0.55, 0.3, 0.15],\n",
" [0.9, 0.2, 0.2],\n",
" [0.2, 0.5, 0.9],\n",
" [0.2, 0.75, 0.3],\n",
" [0.8, 0.3, 0.8],\n",
" [0.95, 0.7, 0.15],\n",
" [0.6, 0.6, 0.6],\n",
" ],\n",
" dtype=np.float32,\n",
")\n",
"\n",
"# Keep every score column and the 16 image rows visible in the tables.\n",
"pl.Config.set_tbl_cols(-1)\n",
"pl.Config.set_tbl_rows(20)\n",
"\n",
"\n",
"def load_prediction(path: Path) -> tuple[np.ndarray, np.ndarray]:\n",
" \"\"\"Read the saved instance and cell-type masks for one image.\"\"\"\n",
" with np.load(path, allow_pickle=False) as saved:\n",
" return saved[\"instances\"], saved[\"classes\"]\n"
],
"execution_count": 1,
"outputs": []
},
{
"cell_type": "markdown",
"id": "metrics-load-heading",
"metadata": {},
"source": [
"## Load matching masks\n",
"\n",
"Instance masks are integer arrays with shape `(height, width)`: zero is background\n",
"and each positive ID identifies one nucleus. IDs need not match between reference\n",
"and prediction. Cell-type masks assign a category to each nucleus.\n",
"\n",
"Both masks must cover the same pixels at the same resolution. No normalization\n",
"or resizing is needed for evaluation. PanopTILs class 0 can contain untyped\n",
"nuclei, so use the **instance mask**, rather than the class mask, to find foreground.\n"
]
},
{
"id": "metrics-compute-heading",
"metadata": {},
"cell_type": "markdown",
"source": [
"## Compute the scores\n",
"\n",
"Panoptic quality (**PQ**) combines two scores:\n",
"\n",
"- **Detection quality (DQ)** measures missed and extra nuclei using object F1.\n",
"- **Segmentation quality (SQ)** is the mean intersection over union (IoU) of\n",
" matched nuclei. IoU measures their shared area divided by their combined area.\n",
"\n",
"`PQ = DQ × SQ`. Scores approach 1 for a perfect result. With `thresh=0.5`, a\n",
"reference nucleus matches a prediction only when their IoU is **greater than**\n",
"0.5. Each nucleus can match at most one other nucleus.\n",
"\n",
"Binary Dice and IoU compare the pixels covered by any nucleus. Merging two nuclei\n",
"can leave foreground coverage unchanged, so use these scores alongside PQ.\n",
"\n",
"The function below evaluates one mask pair and returns one table row. Pairs with\n",
"no nuclei on either side receive null scores, which are excluded from averages.\n",
"An empty mask paired with a nonempty mask scores zero.\n"
]
},
{
"id": "metrics-compute",
"metadata": {},
"cell_type": "code",
"execution_count": 2,
"outputs": [],
"source": [
"def score_masks(reference: np.ndarray, prediction: np.ndarray) -> pl.DataFrame:\n",
" \"\"\"Return nucleus counts and instance/foreground scores for one mask pair.\"\"\"\n",
" quality = panoptic_quality(reference, prediction, thresh=0.5)\n",
" tp, fp, fn = get_stats(\n",
" (reference > 0).astype(np.uint8),\n",
" (prediction > 0).astype(np.uint8),\n",
" )\n",
" result = pl.DataFrame(\n",
" {\n",
" \"Reference nuclei\": [np.count_nonzero(np.unique(reference))],\n",
" \"Predicted nuclei\": [np.count_nonzero(np.unique(prediction))],\n",
" \"PQ\": [quality[\"pq\"]],\n",
" \"DQ\": [quality[\"dq\"]],\n",
" \"SQ\": [quality[\"sq\"]],\n",
" \"Dice\": [dice_coef(tp, fp, fn)],\n",
" \"IoU\": [iou_score(tp, fp, fn)],\n",
" }\n",
" )\n",
"\n",
" # Keep scores undefined only when neither mask contains a nucleus.\n",
" return result.with_columns(\n",
" pl.when(pl.col(\"Reference nuclei\") + pl.col(\"Predicted nuclei\") > 0)\n",
" .then(pl.col(\"PQ\", \"DQ\", \"SQ\", \"Dice\", \"IoU\"))\n",
" .otherwise(None)\n",
" )\n"
]
},
{
"cell_type": "markdown",
"id": "metrics-images-heading",
"metadata": {},
"source": [
"## Evaluate the saved predictions\n",
"\n",
"Read each prediction and its matching labels once. First score all nuclei,\n",
"including untyped ones, then score each cell type separately. The metric functions\n",
"work on NumPy masks; Polars collects and summarizes their scores. Only small score\n",
"tables are kept in memory between images.\n"
]
},
{
"cell_type": "code",
"id": "metrics-images",
"metadata": {},
"source": [
"prediction_paths = sorted(prediction_root.glob(\"*.npz\"))\n",
"tables = []\n",
"\n",
"for path in prediction_paths:\n",
" reference = FileHandler.read_mat(\n",
" data_root / \"validation/regions/labels\" / f\"{path.stem}.mat\"\n",
" )\n",
" instances, classes = load_prediction(path)\n",
"\n",
" per_type = [\n",
" score_masks(reference[\"inst\"], instances).with_columns(\n",
" pl.lit(\"all nuclei\").alias(\"Cell type\")\n",
" )\n",
" ]\n",
" for class_id, name in enumerate(class_names[1:], start=1):\n",
" # Keep original nucleus IDs while selecting one cell type.\n",
" true_class = np.where(reference[\"type\"] == class_id, reference[\"inst\"], 0)\n",
" predicted_class = np.where(classes == class_id, instances, 0)\n",
" per_type.append(\n",
" score_masks(true_class, predicted_class).with_columns(\n",
" pl.lit(name).alias(\"Cell type\")\n",
" )\n",
" )\n",
"\n",
" tables.append(pl.concat(per_type).with_columns(pl.lit(path.stem).alias(\"Image\")))\n",
"\n",
"scores = pl.concat(tables).select(\n",
" \"Image\", \"Cell type\", pl.exclude(\"Image\", \"Cell type\")\n",
")\n",
"image_scores = scores.filter(pl.col(\"Cell type\") == \"all nuclei\").drop(\"Cell type\")\n",
"image_scores.with_columns(pl.col(pl.Float64).round(3))\n"
],
"execution_count": 3,
"outputs": [
{
"data": {
"text/html": [
"\n",
"
shape: (16, 8) Image Reference nuclei Predicted nuclei PQ DQ SQ Dice IoU str i64 i64 f64 f64 f64 f64 f64 "region_0000" 568 623 0.686 0.883 0.777 0.851 0.741 "region_0331" 402 457 0.605 0.78 0.776 0.81 0.68 "region_0386" 274 208 0.594 0.759 0.783 0.81 0.681 "region_0401" 367 317 0.631 0.822 0.768 0.82 0.695 "region_0555" 422 426 0.625 0.823 0.76 0.828 0.706 "region_0604" 587 593 0.659 0.846 0.78 0.857 0.75 "region_0658" 452 471 0.596 0.821 0.726 0.838 0.721 "region_0690" 348 365 0.5 0.662 0.755 0.771 0.627 "region_0804" 487 634 0.564 0.737 0.765 0.732 0.577 "region_0881" 156 180 0.68 0.851 0.799 0.816 0.689 "region_0896" 382 418 0.707 0.877 0.805 0.859 0.754 "region_1126" 237 281 0.718 0.896 0.802 0.848 0.736 "region_1199" 515 619 0.64 0.813 0.787 0.824 0.701 "region_1230" 164 166 0.667 0.855 0.781 0.837 0.72 "region_1252" 1039 1087 0.711 0.907 0.784 0.879 0.784 "region_1267" 356 347 0.702 0.888 0.791 0.84 0.724
"
]
},
"metadata": {},
"output_type": "display_data"
}
]
},
{
"cell_type": "markdown",
"id": "metrics-instance-heading",
"metadata": {},
"source": [
"## Inspect the first region's scores\n",
"\n",
"Select the first illustrated region from the score table. These instance scores\n",
"evaluate nuclei regardless of cell type. The counts help distinguish missed or\n",
"extra nuclei from inaccurate boundaries.\n"
]
},
{
"cell_type": "code",
"id": "metrics-instance",
"metadata": {},
"source": [
"first_region = image_scores.filter(pl.col(\"Image\") == example_images[0])\n",
"quality = first_region.select(\"Reference nuclei\", \"Predicted nuclei\", \"PQ\", \"DQ\", \"SQ\")\n",
"quality.with_columns(pl.col(pl.Float64).round(3))\n"
],
"execution_count": 5,
"outputs": [
{
"data": {
"text/html": [
"\n",
"
shape: (1, 5) Reference nuclei Predicted nuclei PQ DQ SQ i64 i64 f64 f64 f64 1039 1087 0.711 0.907 0.784
\n",
"
shape: (12, 7) Image Cell type Reference nuclei Predicted nuclei PQ DQ SQ str str i64 i64 f64 f64 f64 "region_1252" "neoplastic" 0 0 null null null "region_1252" "stromal" 309 358 0.493 0.654 0.754 "region_1252" "inflammatory" 725 729 0.658 0.825 0.798 "region_1252" "epithelial" 0 0 null null null "region_1252" "other" 0 0 null null null "region_1252" "unknown" 1 0 0.0 0.0 0.0 "region_0804" "neoplastic" 125 90 0.498 0.651 0.766 "region_0804" "stromal" 164 51 0.158 0.214 0.739 "region_0804" "inflammatory" 193 38 0.216 0.268 0.806 "region_0804" "epithelial" 0 0 null null null "region_0804" "other" 0 0 null null null "region_0804" "unknown" 3 455 0.01 0.013 0.742
"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"def cell_type_overlay(\n",
" image: np.ndarray, instances: np.ndarray, classes: np.ndarray\n",
") -> np.ndarray:\n",
" \"\"\"Overlay cell types and nucleus outlines on a uint8 RGB image.\n",
"\n",
" Returns an RGB float32 image in [0, 1], leaving background tissue unchanged.\n",
" \"\"\"\n",
" overlay = image.astype(np.float32) / 255\n",
" colours = class_palette[classes]\n",
" foreground = instances > 0\n",
"\n",
" # A light fill keeps the tissue visible; outlines separate touching nuclei.\n",
" overlay[foreground] = 0.75 * overlay[foreground] + 0.25 * colours[foreground]\n",
" outlines = find_boundaries(instances, mode=\"inner\")\n",
" overlay[outlines] = colours[outlines]\n",
" return overlay\n",
"\n",
"\n",
"overlay_fig, overlay_axes = plt.subplots(\n",
" len(example_images), 3, figsize=(15, 5 * len(example_images)), squeeze=False\n",
")\n",
"\n",
"for row, name in enumerate(example_images):\n",
" image_path = data_root / \"validation/regions/images\" / f\"{name}.png\"\n",
" region_image = cv2.cvtColor(cv2.imread(str(image_path)), cv2.COLOR_BGR2RGB)\n",
" reference = FileHandler.read_mat(\n",
" data_root / \"validation/regions/labels\" / f\"{name}.mat\"\n",
" )\n",
" instances, classes = load_prediction(prediction_root / f\"{name}.npz\")\n",
"\n",
" overlay_axes[row, 0].imshow(region_image)\n",
" overlay_axes[row, 0].set_title(f\"{name}: image\")\n",
"\n",
" overlay_axes[row, 1].imshow(\n",
" cell_type_overlay(region_image, reference[\"inst\"], reference[\"type\"])\n",
" )\n",
" overlay_axes[row, 1].set_title(\"Reference cell types\")\n",
"\n",
" overlay_axes[row, 2].imshow(cell_type_overlay(region_image, instances, classes))\n",
" overlay_axes[row, 2].set_title(\"Predicted cell types\")\n",
"\n",
"for axis in overlay_axes.flat:\n",
" axis.set(xticks=[], yticks=[])\n",
"\n",
"overlay_fig.legend(\n",
" [Patch(color=colour) for colour in class_palette],\n",
" class_names,\n",
" loc=\"lower center\",\n",
" ncol=7,\n",
")\n",
"overlay_fig.tight_layout(rect=(0, 0.04, 1, 1))\n",
"plt.show()\n"
]
},
{
"cell_type": "markdown",
"id": "metrics-summary-heading",
"metadata": {},
"source": [
"## Summarize the results\n",
"\n",
"Average each score across images with a defined value, giving each image equal\n",
"weight. Sum the nucleus counts. These are **means of per-image scores**, not\n",
"scores computed from pooled object matches or pixels. Null entries are excluded;\n",
"missed or extra nuclei with a zero score remain included. Average PQ directly;\n",
"multiplying mean DQ by mean SQ does not give the same result.\n",
"\n",
"For each cell type, “Scored images” counts regions containing that type in the\n",
"reference, the prediction, or both. Keep these counts beside the scores so rare\n",
"categories are not mistaken for well-tested ones.\n"
]
},
{
"cell_type": "code",
"id": "metrics-summary",
"metadata": {},
"source": [
"summary = image_scores.select(\n",
" pl.len().alias(\"Images\"),\n",
" pl.col(\"PQ\").count().alias(\"Scored images\"),\n",
" pl.col(\"Reference nuclei\", \"Predicted nuclei\").sum(),\n",
" pl.col(\"PQ\", \"DQ\", \"SQ\", \"Dice\", \"IoU\").mean(),\n",
")\n",
"summary.with_columns(pl.col(pl.Float64).round(3))\n"
],
"execution_count": 9,
"outputs": [
{
"data": {
"text/html": [
"\n",
"
shape: (1, 9) Images Scored images Reference nuclei Predicted nuclei PQ DQ SQ Dice IoU u32 u32 i64 i64 f64 f64 f64 f64 f64 16 16 6756 7192 0.643 0.826 0.777 0.826 0.705
\n",
"
shape: (6, 7) Cell type Scored images Reference nuclei Predicted nuclei PQ DQ SQ str u32 i64 i64 f64 f64 f64 "neoplastic" 15 3082 2860 0.509 0.659 0.716 "stromal" 16 1740 1986 0.361 0.476 0.66 "inflammatory" 16 1202 1115 0.308 0.375 0.511 "epithelial" 0 0 0 null null null "other" 1 3 0 0.0 0.0 0.0 "unknown" 15 657 1231 0.219 0.279 0.464