Metrics#

Compare predicted masks with reference labels. Use NumPy arrays on CPU: instance masks for individual cells, class masks for categories, or binary masks for foreground/background. Convert raw model scores to labels first.

Instance evaluation#

Use 2D integer masks with 0 for background and a positive ID for each object. IDs can differ between prediction and reference masks. Panoptic quality matches objects by their overlap and returns pq, sq and dq.

import numpy as np

from cellseg_models_pytorch.metrics import panoptic_quality

truth = np.array([[0, 5, 5], [0, 0, 0]], dtype=np.int32)
prediction = np.array([[0, 12, 12], [0, 0, 0]], dtype=np.int32)
score = panoptic_quality(truth, prediction)

score["pq"] is nearly 1 because the object regions match. These instance scores return zero when either mask is empty, including when both are empty. Each call evaluates one image pair.

Function

Meaning and matching

panoptic_quality

pq = dq * sq; dq measures missed and extra objects (F1); sq measures how closely matched objects overlap (IoU)

average_precision

Fraction of predicted objects matched at one overlap threshold; not average precision across a curve

aggregated_jaccard_index

Best-overlapping prediction per reference object; matches can be reused and thresh is ignored

dice2

Dice over overlapping object pairs; ignores pairs with no overlap

For scores by cell type, evaluate instance masks separately for each class. See the evaluation guide for an example.

panoptic_quality(true, pred, thresh=0.5, eps=1e-08)#

Compute panoptic quality with one-to-one instance matching.

Below 0.5 IoU, maximize the number of qualifying pairs, then their summed IoU. Lowering the threshold can increase detection quality while decreasing panoptic quality because pair count takes priority over summed IoU. At or above 0.5, qualifying pairs are already unique.

Parameters:

  • true (ndarray) – Integer instance mask (H, W), with zero as background.

  • pred (ndarray) – Integer instance mask (H, W), with zero as background. Instance IDs need not be contiguous or match the ground truth IDs.

  • thresh (float) – IoU must be strictly greater than this value in [0, 1].

  • eps (float) – Epsilon to avoid zero div errors. Defaults to 1e-8.

Returns:

Dictionary with pq (panoptic quality), sq (mean matched-object overlap) and dq (object detection F1). All three are zero if either mask has no foreground. The function evaluates one mask pair without cell-type classification or dataset aggregation.

Raises:

ValueError – If thresh is outside [0, 1] or not finite.

Return type:

dict[str, float]

Examples

>>> mask = np.array([[0, 1, 1]])
>>> round(float(panoptic_quality(mask, mask)["pq"]), 6)
1.0
average_precision(true, pred, thresh=0.5, eps=1e-08)#

Compute object precision, TP / (TP + FP), at one IoU threshold.

Despite the historical name, this is not area under a precision-recall curve. Match instances uniquely, maximizing qualifying pair count then IoU.

Parameters:

  • true (ndarray) – Integer instance mask (H, W), with zero as background.

  • pred (ndarray) – Integer instance mask (H, W), with zero as background. Instance IDs need not be contiguous or match the ground truth IDs.

  • thresh (float) – IoU must be strictly greater than this value in [0, 1].

  • eps (float) – Epsilon to avoid zero div errors. Defaults to 1e-8.

Returns:

Matched prediction fraction at this threshold. Zero if either mask has no foreground. This is object precision, not TP / (TP + FP + FN), and does not integrate a confidence-ranked precision-recall curve.

Raises:

ValueError – If thresh is outside [0, 1] or not finite.

Return type:

float

Examples

>>> mask = np.array([[0, 1, 1]])
>>> round(average_precision(mask, mask), 6)
1.0
aggregated_jaccard_index(true, pred, thresh=0.5, eps=1e-08)#

Aggregate instance overlap using each true object’s largest intersection.

Parameters:

  • true (ndarray) – Integer HW instance mask; 0 is background and positive IDs are objects.

  • pred (ndarray) – Same-shaped prediction mask. IDs may be nonconsecutive and unrelated to true IDs.

  • thresh (float) – Retained argument, currently unused. It does not filter matches.

  • eps (float) – Small denominator offset.

Returns:

Scalar intersection/union aggregate. Matching uses largest intersection area per true object, not largest IoU. Ties can select several predictions and the same prediction can be selected for several true objects. Unmatched objects contribute their area to the denominator. Returns zero if either mask has no foreground. This is not unique AJI+ matching.

Return type:

float

dice2(true, pred, eps=1e-08)#

Compute the package’s pairwise instance Dice aggregate.

Parameters:

  • true (ndarray) – Integer HW instance mask; 0 is background, positive IDs are objects.

  • pred (ndarray) – Same-shaped prediction mask. Object IDs need not match true IDs.

  • eps (float) – Small denominator offset.

Returns:

Scalar from summed pairwise intersections and fp/fn entries for overlapping object pairs. This does not use one-to-one object matching; objects with no overlapping pair do not contribute to these matrices. Returns zero if either mask has no foreground. Do not interpret it as ordinary binary Dice on the two foreground masks.

Return type:

float

Binary pixel evaluation#

Pass 0/1 masks to get_stats, then pass its three result arrays to a score function. Keep these arrays per pixel when using accuracy.

from cellseg_models_pytorch.metrics import get_stats, iou_score

binary_truth = (truth > 0).astype(np.uint8)
binary_prediction = (prediction > 0).astype(np.uint8)
tp, fp, fn = get_stats(binary_truth, binary_prediction)
foreground_iou = iou_score(tp, fp, fn)
get_stats(true, pred)#

Mark true-positive, false-positive and false-negative pixels.

Parameters:

  • true (ndarray) – HW binary NumPy mask with integer or boolean values 0 and 1.

  • pred (ndarray) – Same-shaped binary prediction mask. Do not supply instance IDs, class IDs above 1 or floating probabilities.

Returns:

(tp, fp, fn) per-pixel indicator arrays of the input shape. Each marks the relevant pixels with 1 (or True), and others with 0. They are not already-summed counts; pass them to the binary score functions.

Return type:

Tuple[ndarray, ndarray, ndarray]

Example

>>> true = np.array([[0, 1, 1]], dtype=np.uint8)
>>> pred = np.array([[0, 1, 0]], dtype=np.uint8)
>>> [int(x.sum()) for x in get_stats(true, pred)]
[1, 0, 1]
iou_score(tp, fp, fn, eps=1e-08)#

Compute binary intersection over union from pixel statistics.

The score sums the supplied indicator arrays before computing the ratio.

Parameters:

  • tp (ndarray) – True-positive indicator array from get_stats.

  • fp (ndarray) – False-positive indicator array of the same shape.

  • fn (ndarray) – False-negative indicator array of the same shape.

  • eps (float) – Small value added to the denominator to avoid division by zero.

Returns:

Scalar intersection over union, using TP / (TP + FP + FN) with the stated epsilon. All-background pairs score zero for overlap/F1 and nearly one for accuracy. No thresholding, instance matching or dataset aggregation is performed.

Return type:

float

dice_coef(tp, fp, fn, eps=1e-08)#

Compute binary Dice overlap from pixel statistics.

The score sums the supplied indicator arrays before computing the ratio.

Parameters:

  • tp (ndarray) – True-positive indicator array from get_stats.

  • fp (ndarray) – False-positive indicator array of the same shape.

  • fn (ndarray) – False-negative indicator array of the same shape.

  • eps (float) – Small value added to the denominator to avoid division by zero.

Returns:

Scalar Dice overlap, using 2 * TP / (2 * TP + FP + FN) with the stated epsilon. All-background pairs score zero for overlap/F1 and nearly one for accuracy. No thresholding, instance matching or dataset aggregation is performed.

Return type:

float

accuracy(tp, fp, fn, eps=1e-08)#

Compute binary pixel accuracy from pixel statistics.

Accuracy derives true negatives from per-pixel 0/1 indicators, so do not supply aggregate counts.

Parameters:

  • tp (ndarray) – True-positive indicator array from get_stats.

  • fp (ndarray) – False-positive indicator array of the same shape.

  • fn (ndarray) – False-negative indicator array of the same shape.

  • eps (float) – Small value added to the denominator to avoid division by zero.

Returns:

Scalar pixel accuracy, using (TP + TN) / number_of_pixels with the stated epsilon. All-background pairs score zero for overlap/F1 and nearly one for accuracy. No thresholding, instance matching or dataset aggregation is performed.

Return type:

float

f1score(tp, fp, fn, eps=1e-08)#

Compute binary F1 score from pixel statistics.

The score sums the supplied indicator arrays before computing the ratio.

Parameters:

  • tp (ndarray) – True-positive indicator array from get_stats.

  • fp (ndarray) – False-positive indicator array of the same shape.

  • fn (ndarray) – False-negative indicator array of the same shape.

  • eps (float) – Small value added to the denominator to avoid division by zero.

Returns:

Scalar F1 score, using TP / (TP + 0.5 * FP + 0.5 * FN) with the stated epsilon. All-background pairs score zero for overlap/F1 and nearly one for accuracy. No thresholding, instance matching or dataset aggregation is performed.

Return type:

float

Class-mask evaluation#

Use 2D class masks with labels from 0 to num_classes - 1. Results contain one score per class, including class 0. With clamp_absent=True, a class absent from both masks gets -1; leave those values out of averages. Classes present in only one mask are still evaluated.

specificity_multiclass measures how often pixels outside a reference class are correctly excluded from that predicted class: TN / (TN + FP).

iou_multiclass(true, pred, num_classes, eps=1e-08, clamp_absent=True)#

Compute intersection over union (IoU) for each class in a pair of masks.

Parameters:

  • true (ndarray) – Integer HW class mask with IDs from 0 to num_classes-1.

  • pred (ndarray) – Same-shaped predicted class IDs, not probabilities or instance IDs.

  • num_classes (int) – Total class count, including class 0 if it is background.

  • eps (float) – Small offset added to the denominator.

  • clamp_absent (bool) – Mark a class absent from both masks as -1. False retains the computed value. Absent-class markers must be excluded from means.

Returns:

Float64 array (num_classes,), ordered by class ID, including class 0. Uses TP / (TP + FP + FN) from the current pairwise statistics. No batch or dataset mean is calculated. H is height and W width.

Return type:

ndarray

dice_multiclass(true, pred, num_classes, eps=1e-08, clamp_absent=True)#

Compute Dice overlap for each class in a pair of class masks.

Parameters:

  • true (ndarray) – Integer HW class mask with IDs from 0 to num_classes-1.

  • pred (ndarray) – Same-shaped predicted class IDs, not probabilities or instance IDs.

  • num_classes (int) – Total class count, including class 0 if it is background.

  • eps (float) – Small offset added to the denominator.

  • clamp_absent (bool) – Mark a class absent from both masks as -1. False retains the computed value. Absent-class markers must be excluded from means.

Returns:

Float64 array (num_classes,), ordered by class ID, including class 0. Uses 2 * TP / (2 * TP + FP + FN) from the current pairwise statistics. No batch or dataset mean is calculated. H is height and W width.

Return type:

ndarray

f1score_multiclass(true, pred, num_classes, eps=1e-08, clamp_absent=True)#

Compute F1 for each class in a pair of class masks.

Parameters:

  • true (ndarray) – Integer HW class mask with IDs from 0 to num_classes-1.

  • pred (ndarray) – Same-shaped predicted class IDs, not probabilities or instance IDs.

  • num_classes (int) – Total class count, including class 0 if it is background.

  • eps (float) – Small offset added to the denominator.

  • clamp_absent (bool) – Mark a class absent from both masks as -1. False retains the computed value. Absent-class markers must be excluded from means.

Returns:

Float64 array (num_classes,), ordered by class ID, including class 0. Uses TP / (TP + 0.5 * FP + 0.5 * FN) from the current pairwise statistics. No batch or dataset mean is calculated. H is height and W width.

Return type:

ndarray

accuracy_multiclass(true, pred, num_classes, eps=1e-08, clamp_absent=True)#

Compute pixel accuracy for each class against all other classes.

Parameters:

  • true (ndarray) – Integer HW class mask with IDs from 0 to num_classes-1.

  • pred (ndarray) – Same-shaped predicted class IDs, not probabilities or instance IDs.

  • num_classes (int) – Total class count, including class 0 if it is background.

  • eps (float) – Small offset added to the denominator.

  • clamp_absent (bool) – Mark a class absent from both masks as -1. False retains the computed value. Absent-class markers must be excluded from means.

Returns:

Float64 array (num_classes,), ordered by class ID, including class 0. Uses (TP + TN) / pixel_count from the current pairwise statistics. No batch or dataset mean is calculated. H is height and W width.

Return type:

ndarray

sensitivity_multiclass(true, pred, num_classes, eps=1e-08, clamp_absent=True)#

Compute sensitivity (recall) for each class in a pair of class masks.

Parameters:

  • true (ndarray) – Integer HW class mask with IDs from 0 to num_classes-1.

  • pred (ndarray) – Same-shaped predicted class IDs, not probabilities or instance IDs.

  • num_classes (int) – Total class count, including class 0 if it is background.

  • eps (float) – Small offset added to the denominator.

  • clamp_absent (bool) – Mark a class absent from both masks as -1. False retains the computed value. Absent-class markers must be excluded from means.

Returns:

Float64 array (num_classes,), ordered by class ID, including class 0. Uses TP / (TP + FN) from the current pairwise statistics. No batch or dataset mean is calculated. H is height and W width.

Return type:

ndarray

specificity_multiclass(true, pred, num_classes, eps=1e-08, clamp_absent=True)#

Compute specificity for each class against all other classes.

Parameters:

  • true (ndarray) – Integer HW class mask with IDs from 0 to num_classes-1.

  • pred (ndarray) – Same-shaped predicted class IDs, not probabilities or instance IDs.

  • num_classes (int) – Total class count, including class 0 if it is background.

  • eps (float) – Small offset added to the denominator.

  • clamp_absent (bool) – Mark a class absent from both masks as -1. False retains the computed value. Absent-class markers must be excluded from means.

Returns:

Float64 array (num_classes,), ordered by class ID, including class 0. Uses TN / (TN + FP) from the pairwise statistics. No batch or dataset mean is calculated. H is height and W width.

Return type:

ndarray

Pairwise statistics#

Use these helpers for counts between object or class pairs. Class-mode pixel counts include missed and extra pixels even when a class has no correct overlap. Instance mode covers only overlapping object pairs. Object counts need a match matrix with at most one match per object; the helper does not choose matches.

pairwise_pixel_stats(true, pred, num_classes=None, metric_func=None)#

Build overlap statistics for ground-truth and prediction label pairs.

Parameters:

  • true (ndarray) – Integer HW NumPy mask. In instance mode, 0 is background and positive IDs identify objects; IDs need not be consecutive.

  • pred (ndarray) – Same-shaped integer prediction mask.

  • num_classes (int | None) – None compares sorted nonzero instance IDs. An integer compares class IDs 0 through num_classes-1, including class 0.

  • metric_func (Callable[[ndarray, ndarray, ndarray], float] | None) – Optional function accepting tp, fp, fn pixel indicators, for example iou_score, to produce one value per class pair or overlapping instance pair.

Returns:

List of float64 pairwise matrices. Without metric_func, returns [tp, fp, fn] counts; with it, a one-element list containing that score. Rows correspond to sorted true IDs and columns to sorted predicted IDs, or to all class IDs in class mode. Class mode counts fp/fn even for non-overlapping pairs, including absent classes. Instance mode leaves non-overlapping pairs at zero and returns an empty list if either mask has no foreground.

Return type:

list[ndarray]

pairwise_object_stats(matches, sum_reduce=True)#

Count detected, spurious and missed objects from a match matrix.

Parameters:

  • matches (ndarray) – Boolean matrix (true_objects, predicted_objects). Supply one-to-one matches when counting true-positive pairs; this helper does not resolve ambiguous assignments.

  • sum_reduce (bool) – True returns totals. False returns the indicator arrays.

Returns:

(tp, fp, fn) integers, or boolean arrays when sum_reduce=False. TP and FN arrays index ground-truth rows; FP indexes prediction columns. TP counts matched rows rather than every True matrix element.

Return type:

tuple[int, int, int] | tuple[ndarray, ndarray, ndarray]