{
"cells": [
{
"cell_type": "markdown",
"id": "bea7d7fd",
"metadata": {},
"source": [
"# Prepare training data\n",
"\n",
"Prepare image patches and matching nucleus and cell-type labels from\n",
"[PanopTILs Refined](https://huggingface.co/datasets/csmp-hub/panoptils_refined)\n",
"for training. An instance label identifies one nucleus; a class label describes\n",
"its type. The dataset is CC0-1.0 and builds on PanopTILs, BCSS and NuCLS.\n",
"\n",
"Download the data, separate patients into training and validation sets, then crop\n",
"images and labels together. Only nucleus instances and cell types are used here.\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "cb07201d",
"metadata": {},
"outputs": [],
"source": [
"import json\n",
"import random\n",
"from pathlib import Path\n",
"from tempfile import TemporaryDirectory\n",
"\n",
"import cv2\n",
"import matplotlib.pyplot as plt\n",
"import numpy as np\n",
"import polars as pl\n",
"from huggingface_hub import hf_hub_download\n",
"from matplotlib.patches import Patch\n",
"from scipy.io import loadmat, savemat\n",
"from skimage.color import label2rgb\n",
"from skimage.measure import regionprops\n",
"\n",
"dataset_id = \"csmp-hub/panoptils_refined\"\n",
"revision = \"21cc3a3985117c84fb399f0048920eba42fbba32\"\n",
"archive_sha256 = \"1ab3f0c2299e0c44f996e34c233846592a934aa3dcfc8e80cc983db85c771d18\"\n",
"patch_size = 256\n",
"seed = 42\n",
"data_root = Path(\"../../data/panoptils-nuclei\")\n",
"class_names = [\n",
" \"background\",\n",
" \"neoplastic\",\n",
" \"stromal\",\n",
" \"inflammatory\",\n",
" \"epithelial\",\n",
" \"other\",\n",
" \"unknown\",\n",
"]\n"
]
},
{
"cell_type": "markdown",
"id": "a074dbc6",
"metadata": {},
"source": [
"## Download the dataset\n",
"\n",
"Download the complete archive from CSMP Hub. This walkthrough creates its own\n",
"patient-separated training and validation split below, rather than using the\n",
"Hub's separate split files. Keep the same version throughout preparation and training.\n"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "4d4ee63c",
"metadata": {},
"outputs": [],
"source": [
"archive = Path(\n",
" hf_hub_download(\n",
" repo_id=dataset_id,\n",
" repo_type=\"dataset\",\n",
" filename=\"panoptils_refined.parquet\",\n",
" revision=revision,\n",
" )\n",
")\n"
]
},
{
"cell_type": "markdown",
"id": "75b00553",
"metadata": {},
"source": [
"## Split by patient\n",
"\n",
"Keep each patient's images in one split so validation checks the model on new\n",
"patients. Use about 80% of patients for training and 20% for validation.\n"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "291b9440",
"metadata": {},
"outputs": [],
"source": [
"dataset = pl.scan_parquet(archive).with_row_index(\"row\")\n",
"# Load only the small metadata table; image bytes stay on disk.\n",
"regions = (\n",
" dataset.select(\"row\", \"slide_name\")\n",
" .with_columns(\n",
" pl.col(\"slide_name\")\n",
" .str.split(\"-\")\n",
" .list.slice(0, 3)\n",
" .list.join(\"-\")\n",
" .alias(\"patient\")\n",
" )\n",
" .collect()\n",
")\n",
"\n",
"# Slides from one patient share the first three TCGA identifier fields.\n",
"patients = sorted(regions[\"patient\"].unique().to_list())\n",
"random.Random(seed).shuffle(patients)\n",
"validation_patients = patients[: max(1, len(patients) // 5)]\n",
"\n",
"regions = regions.with_columns(\n",
" pl.when(pl.col(\"patient\").is_in(validation_patients))\n",
" .then(pl.lit(\"validation\"))\n",
" .otherwise(pl.lit(\"train\"))\n",
" .alias(\"split\"),\n",
" pl.format(\"region_{}\", pl.col(\"row\").cast(pl.String).str.zfill(4)).alias(\"file\"),\n",
")\n"
]
},
{
"cell_type": "markdown",
"id": "f57c4ec2",
"metadata": {},
"source": [
"## Read images and labels\n",
"\n",
"Each region contains an RGB image, a nucleus-instance mask and a cell-type mask.\n",
"The masks must line up with the image. Zero in the instance mask is background;\n",
"positive numbers identify nuclei.\n"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "5822f6f3",
"metadata": {},
"outputs": [],
"source": [
"def decode_png(data: bytes) -> np.ndarray:\n",
" # Keep the original integer precision of the label masks.\n",
" return cv2.imdecode(np.frombuffer(data, dtype=np.uint8), cv2.IMREAD_UNCHANGED)\n",
"\n",
"\n",
"def decode_region(\n",
" image: bytes, instances: bytes, cell_types: bytes\n",
") -> tuple[np.ndarray, np.ndarray, np.ndarray]:\n",
" image = decode_png(image)\n",
" instances = decode_png(instances)\n",
" cell_types = decode_png(cell_types)\n",
"\n",
" # OpenCV reads BGR; plots and model inputs use RGB.\n",
" image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n",
" return image, instances, cell_types\n"
]
},
{
"cell_type": "markdown",
"id": "90a855f4",
"metadata": {},
"source": [
"## Count nuclei by cell type\n",
"\n",
"Count each nucleus once in its original region, before cropping. This avoids\n",
"counting it again when it crosses a patch boundary. Its type is the most common\n",
"nonzero label inside its mask. A nucleus with no nonzero type label is “untyped”.\n"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "aff0a482",
"metadata": {},
"outputs": [],
"source": [
"def count_nuclei(instances: np.ndarray, cell_types: np.ndarray) -> np.ndarray:\n",
" counts = np.zeros(len(class_names), dtype=np.int64)\n",
" for nucleus in regionprops(instances):\n",
" # Inspect only this nucleus inside its bounding box.\n",
" labels = cell_types[nucleus.slice][nucleus.image]\n",
" votes = np.bincount(labels, minlength=len(class_names))\n",
" votes[0] = 0 # Ignore background pixels when choosing the cell type.\n",
" counts[votes.argmax()] += 1\n",
" return counts\n"
]
},
{
"cell_type": "markdown",
"id": "47a82688",
"metadata": {},
"source": [
"## Save matching images and masks\n",
"\n",
"Save each image as a PNG and its two masks in a MAT file with the same name.\n",
"The MAT file keeps the nucleus IDs in `inst` and the cell types in `type`.\n"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "9c0f5ddd",
"metadata": {},
"outputs": [],
"source": [
"def save_sample(\n",
" folder: Path,\n",
" name: str,\n",
" image: np.ndarray,\n",
" instances: np.ndarray,\n",
" cell_types: np.ndarray,\n",
") -> None:\n",
" image_path = folder / \"images\" / f\"{name}.png\"\n",
" # Convert RGB back to OpenCV's BGR order before saving.\n",
" if not cv2.imwrite(str(image_path), cv2.cvtColor(image, cv2.COLOR_RGB2BGR)):\n",
" raise OSError(f\"Could not write {image_path}\")\n",
" savemat(\n",
" folder / \"labels\" / f\"{name}.mat\",\n",
" {\"inst\": instances.astype(np.int32), \"type\": cell_types},\n",
" do_compression=True,\n",
" )\n"
]
},
{
"cell_type": "markdown",
"id": "3af21a0b",
"metadata": {},
"source": [
"## Create the patches\n",
"\n",
"Cut each 1024-pixel region into sixteen 256-pixel patches, cropping images and\n",
"masks together. Cropping keeps nuclei at their original size. Keep whole validation\n",
"images as well so you can inspect complete predictions later.\n",
"\n",
"Choose a new `data_root` folder for preparation. Reuse it for training.\n"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "af12ac40",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"train: 1096 regions, 17536 patches\n",
"validation: 253 regions, 4048 patches\n"
]
}
],
"source": [
"if data_root.exists():\n",
" raise FileExistsError(f\"Choose a new data_root folder: {data_root}\")\n",
"\n",
"counts = {\n",
" split: {\"regions\": 0, \"patches\": 0, \"class_instances\": [0] * len(class_names)}\n",
" for split in (\"train\", \"validation\")\n",
"}\n",
"data_root.parent.mkdir(parents=True, exist_ok=True)\n",
"\n",
"\n",
"# Publish the prepared folder only after every region has been saved.\n",
"with TemporaryDirectory(dir=data_root.parent, prefix=\".panoptils-\") as temporary:\n",
" root = Path(temporary)\n",
" for folder in (root / \"train\", root / \"validation\", root / \"validation/regions\"):\n",
" (folder / \"images\").mkdir(parents=True)\n",
" (folder / \"labels\").mkdir()\n",
"\n",
" # Decode one region at a time; Polars may buffer a larger Parquet row group.\n",
" images = dataset.select(\"row\", \"image\", \"inst\", \"type\")\n",
" for batch in images.collect_batches(chunk_size=1):\n",
" for row in batch.iter_rows(named=True):\n",
" image, instances, cell_types = decode_region(\n",
" row[\"image\"], row[\"inst\"], row[\"type\"]\n",
" )\n",
" record = regions.row(row[\"row\"], named=True)\n",
" split = record[\"split\"]\n",
" region_name = record[\"file\"]\n",
"\n",
" counts[split][\"regions\"] += 1\n",
" counts[split][\"class_instances\"] = (\n",
" np.array(counts[split][\"class_instances\"])\n",
" + count_nuclei(instances, cell_types)\n",
" ).tolist()\n",
"\n",
" # Keep unpatched validation regions for evaluating whole-image predictions.\n",
" if split == \"validation\":\n",
" save_sample(\n",
" root / \"validation/regions\",\n",
" region_name,\n",
" image,\n",
" instances,\n",
" cell_types,\n",
" )\n",
"\n",
" # Crop both splits into patches, keeping each image and its masks aligned.\n",
" height, width = instances.shape\n",
" for y in range(0, height, patch_size):\n",
" for x in range(0, width, patch_size):\n",
" area = np.s_[y : y + patch_size, x : x + patch_size]\n",
" name = f\"{region_name}_y{y:04d}_x{x:04d}\"\n",
" save_sample(\n",
" root / split,\n",
" name,\n",
" image[area],\n",
" instances[area],\n",
" cell_types[area],\n",
" )\n",
" counts[split][\"patches\"] += 1\n",
"\n",
" manifest = {\n",
" \"dataset\": dataset_id,\n",
" \"revision\": revision,\n",
" \"sha256\": archive_sha256,\n",
" \"patch_size\": patch_size,\n",
" \"seed\": seed,\n",
" \"samples\": regions.select(\"row\", \"patient\", \"split\", \"file\").to_dicts(),\n",
" \"splits\": counts,\n",
" }\n",
" (root / \"manifest.json\").write_text(json.dumps(manifest, indent=2) + \"\\n\")\n",
" if data_root.exists():\n",
" raise FileExistsError(data_root)\n",
"\n",
" root.rename(data_root)\n",
"\n",
"for split, count in counts.items():\n",
" print(f\"{split}: {count['regions']} regions, {count['patches']} patches\")\n"
]
},
{
"cell_type": "markdown",
"id": "9bf0abdd",
"metadata": {},
"source": [
"## Inspect the nuclei and labels\n",
"\n",
"The table counts nuclei, not pixels or cropped fragments. Neoplastic means tumour\n",
"cells; inflammatory means immune cells. Stromal cells support the surrounding\n",
"tissue, and epithelial cells form tissue linings. “Other” and “unknown” are labels\n",
"provided by the dataset. “Untyped” means a nucleus has no nonzero cell-type label.\n",
"\n",
"Compare the image with its nucleus and cell-type labels.\n"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "3f6581da",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"
shape: (7, 3)| Cell type | train | validation |
|---|
| str | i64 | i64 |
| "untyped (class 0)" | 4392 | 935 |
| "neoplastic" | 233199 | 45557 |
| "stromal" | 167019 | 32384 |
| "inflammatory" | 145961 | 19144 |
| "epithelial" | 1148 | 90 |
| "other" | 3727 | 493 |
| "unknown" | 24433 | 7709 |