Choose Protein Classifier Thresholds with Precision and Recall

A protein classifier can reach 92% accuracy by rejecting every candidate when only eight of 100 records are positive. That number says little about whether the model can produce a useful shortlist for experiments.

This tutorial turns prediction scores into a selection rule: choose a threshold on validation data, freeze it, and report what happens on a separate test set. It continues the grouped split tutorial and applies to scores from either the composition baseline or an ESM-2 embedding classifier.

1. Decide what a positive prediction costs

Here, label 1 means a candidate met a predefined assay criterion; label 0 means it did not. Higher scores favor label 1. An untested protein is not automatically a negative example.

TP, FP, and FN mean true positives, false positives, and false negatives. Precision and recall use different denominators; neither replaces the experiment count. See the scikit-learn definitions.

For the example, set a validation precision requirement of 0.60, then maximize recall among eligible thresholds. This is an illustrative policy, not a recommended target for protein screening. Choose the requirement from the assay budget and the cost of missed candidates before evaluating models. A fixed top-k shortlist needs a separately specified rule, including how to handle tied scores.

2. Install and run the example

The example contains invented scores and labels for 100 validation records and 100 separate test records. Each set has eight positives. It trains no model and makes no claim about biological performance. The repeated score grid makes the arithmetic easy to compare; the records are separate by construction.

Download threshold_demo.py. With Python 3.11 or newer, run these commands in a new directory after saving the file there:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install "scikit-learn==1.9.1"
python threshold_demo.py --help
python threshold_demo.py --min-precision 0.60

Tested with Python 3.14.7, NumPy 2.5.3, and scikit-learn 1.9.1. Expected output:

Validation precision requirement: 0.600
validation: threshold=0.730 selected=6/100
  TP=4 FP=2 FN=4 TN=90
  precision=0.667 recall=0.500
  AP=0.628 prevalence=0.080
test (frozen threshold): threshold=0.730 selected=6/100
  TP=3 FP=3 FN=5 TN=89
  precision=0.500 recall=0.375
  AP=0.547 prevalence=0.080
test (predeclared 0.5 reference): threshold=0.500 selected=10/100
  TP=4 FP=6 FN=4 TN=86
  precision=0.400 recall=0.500
  AP=0.547 prevalence=0.080
Always-negative test accuracy: 0.920

The validation rule selects threshold 0.730: six records pass, four are positive, and four positives are missed. On the test set, that same threshold selects three positives and three negatives. Test precision is 0.500, below the validation requirement. A requirement met on validation data is not a guarantee about new candidates.

3. Select on validation data and freeze the rule

Here is the complete downloadable script:

"""Synthetic precision/recall tutorial; no trained model or biological results."""
import argparse

import numpy as np
from sklearn.metrics import average_precision_score, precision_recall_curve


def checked_arrays(labels, scores):
    y = np.asarray(labels)
    s = np.asarray(scores, dtype=float)
    if y.ndim != 1 or s.ndim != 1 or y.size != s.size or not y.size:
        raise ValueError("Labels and scores must be nonempty 1D arrays of equal length")
    if not np.isin(y, [0, 1]).all() or np.unique(y).size != 2:
        raise ValueError("Labels must contain both 0 and 1")
    if not np.isfinite(s).all():
        raise ValueError("Scores must be finite")
    return y.astype(int), s


def choose_threshold(labels, scores, min_precision=0.60):
    y, s = checked_arrays(labels, scores)
    if not np.isfinite(min_precision) or not 0 < min_precision <= 1:
        raise ValueError("min_precision must be in (0, 1]")
    precision, recall, thresholds = precision_recall_curve(y, s, pos_label=1)
    # The final plotting endpoint has no threshold and must not be selected.
    eligible = np.flatnonzero(precision[:-1] >= min_precision)
    if not eligible.size:
        raise ValueError("No observed threshold meets the precision requirement")
    # Maximize recall, then precision, then threshold for a deterministic tie rule.
    best = max(eligible, key=lambda i: (recall[i], precision[i], thresholds[i]))
    return float(thresholds[best])


def report(name, labels, scores, threshold):
    y, s = checked_arrays(labels, scores)
    if not np.isfinite(threshold):
        raise ValueError("Threshold must be finite")
    pred = s >= threshold
    tp = int(np.sum(pred & (y == 1)))
    fp = int(np.sum(pred & (y == 0)))
    fn = int(np.sum(~pred & (y == 1)))
    tn = int(np.sum(~pred & (y == 0)))
    precision = f"{tp / (tp + fp):.3f}" if tp + fp else "undefined"
    print(f"{name}: threshold={threshold:.3f} selected={tp + fp}/{len(y)}")
    print(f"  TP={tp} FP={fp} FN={fn} TN={tn}")
    print(f"  precision={precision} recall={tp / (tp + fn):.3f}")
    print(f"  AP={average_precision_score(y, s):.3f} prevalence={y.mean():.3f}")


def demo_data():
    # Separate invented validation and test records, ordered by decreasing score.
    scores = np.r_[
        [.96, .91, .87, .82, .78, .73, .68, .62, .57, .52,
         .47, .42, .37, .32, .28, .24, .20, .16, .12, .08],
        np.linspace(.07, .001, 80),
    ]
    valid = np.zeros(100, dtype=int)
    test = np.zeros(100, dtype=int)
    valid[np.array([1, 3, 4, 6, 9, 11, 16, 20]) - 1] = 1
    test[np.array([1, 2, 5, 8, 12, 20, 24, 30]) - 1] = 1
    return valid, scores.copy(), test, scores.copy()


def main():
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--min-precision", type=float, default=0.60)
    args = parser.parse_args()
    valid_y, valid_s, test_y, test_s = demo_data()
    try:
        threshold = choose_threshold(valid_y, valid_s, args.min_precision)
    except ValueError as exc:
        parser.error(str(exc))
    print(f"Validation precision requirement: {args.min_precision:.3f}")
    report("validation", valid_y, valid_s, threshold)
    report("test (frozen threshold)", test_y, test_s, threshold)
    report("test (predeclared 0.5 reference)", test_y, test_s, 0.5)
    print(f"Always-negative test accuracy: {(test_y == 0).mean():.3f}")


if __name__ == "__main__":
    main()

choose_threshold() only receives validation labels and scores. It maximizes recall among qualifying thresholds, breaks ties by precision, and finally prefers the larger threshold. Predictions use score >= threshold, so every record tied at the boundary receives the same decision.

precision_recall_curve() returns one more precision/recall pair than thresholds. Its final pair is a plotting endpoint without a matching threshold. The code excludes it with precision[:-1]; otherwise, a selection routine might treat that endpoint as a usable operating point. The API documentation specifies this alignment.

For real data, train the model and fitted preprocessing on training records, choose the threshold on validation predictions, and evaluate the frozen rule on test predictions. Keep related proteins together throughout that process. The threshold tuning guide warns against fitting a classifier and tuning its threshold on the same records.

The script also reports a predeclared 0.5 reference. It selects ten test records, including four positives. This describes a trade-off; it does not authorize switching to 0.5 after seeing test results. If you change the model or rule based on those results, use fresh held-out data for the next final evaluation.

4. Read average precision alongside the shortlist

Average precision, or AP, summarizes ranking across thresholds. This script uses average_precision_score, whose calculation differs from trapezoidal integration of a precision-recall curve. Name the exact metric rather than labeling both calculations “PR-AUC.” See the AP documentation.

Test AP is 0.547 in both reports because the underlying test scores never change. Changing the decision threshold changes the shortlist, not the ranking. The invented example therefore separates two questions: whether positives tend to receive higher scores, and whether a particular selection rule produces a useful batch.

Report the positive fraction alongside AP and precision. Here it is 0.080. The always-negative rule has 0.920 accuracy and zero recall, so its high accuracy would not help find any candidates. Precision from this toy dataset does not estimate hit rates in a candidate pool with a different composition or assay policy.

The values between zero and one in this demo are arbitrary scores, not calibrated probabilities. In a real binary classifier, confirm which column of predict_proba() corresponds to label 1 using model.classes_. Do not assume an embedding coordinate, cosine similarity, or an arbitrary decision score is a probability.

5. Replace the toy arrays carefully

Match each score to its label using stable sample IDs; the pandas join tutorial shows a strict alignment check. Pass validation labels and scores to choose_threshold(), save its returned value, and pass that value unchanged to the test report.

Save the model identifier, dataset and split versions, positive-label definition, score direction, precision requirement, selected threshold, >= comparison rule, and tie policy. Report TP, FP, FN, TN, selected count, and the number of independent groups. Eight positives are enough to explain this code, but do not support a precise estimate of a screening hit rate.

6. Keep the decision reproducible

Choose the operating requirement before inspecting test outcomes, select the threshold on validation predictions, and preserve it with the model. Publish the confusion counts and shortlist size alongside AP so readers can see what the rule actually does.

The cover is conceptual artwork, not an experimental protein structure. Documentation checked September 28, 2026.