nanoBERT and VHHBERT: A Practical Guide to Nanobody Language Models

nanoBERT and VHHBERT learn from nanobody sequences. Both can supply features for downstream models, but their published experiments answer different questions. nanoBERT focuses on plausible amino acid substitutions; VHHBERT was introduced alongside a VHH corpus and an antigen-binding benchmark.

For a first project, I would compare their embeddings on the same data split before fine-tuning either model. A larger training corpus, a larger hidden state, or a better score in a different paper does not decide which will work best on your task.

This guide connects the site’s antibody model evaluation article with a concrete nanobody example. Sources and public checkpoints were checked on September 22, 2026. The code below checks software behavior; it is not a new biological benchmark.

What the two models learn

A VHH is the variable domain of a camelid heavy-chain-only antibody. Both models use masked language modeling: hide residues and learn to recover them from the surrounding sequence. This produces contextual residue representations and, with the language-model head, residue predictions. The nanoBERT repository and VHHBERT model card provide the starting implementations.

The training and model-size figures come from the nanoBERT paper and VHHBERT model card. VHHCorpus-2M contains 2,040,988 sequences in total, split into 2,000,000 for training and 40,988 for validation; see the dataset card.

Do not assume that every checkpoint named nanoBERT is the paper’s larger model. The public nanoBERT configuration specifies six layers and hidden size 320. Our example pins that release. VHHBERT’s released encoder has twelve layers and hidden size 768. These widths describe feature vectors, not prediction accuracy.

What the published evidence supports

The nanoBERT study motivates gene-agnostic modeling because camelid germline references are incomplete. It evaluates masked-residue recovery against human-antibody models and ESM-2, and explores downstream nativeness and thermostability tasks. Its results support testing a nanobody-specific prior when selecting plausible substitutions. They do not turn residue probabilities into antigen-specific affinity measurements. See the original study.

The VHHBERT paper evaluates binding with a complete supervised pipeline: a VHH encoder, a frozen ESM-2 antigen encoder, and a trained classifier. The VHH encoder is fine-tuned. Training and test examples come from different alpacas. VHHBERT reaches F1 0.608 ± 0.012 and AUPRC 0.650 ± 0.025; AntiBERTa2-CSSP is higher on both metrics in that experiment. These are reported results over five seeds, not results from the example below. See Table 4 and the experimental setup.

That comparison evaluates transfer to VHHs from another animal within the benchmark’s antigen collection. It does not establish performance on arbitrary new antigens. Nor does it provide a nanoBERT-versus-VHHBERT ranking: nanoBERT is absent from that table.

Match the output to the question

I would use the following starting points, then choose with held-out data:

An embedding is a vector, not a measurement. A residue score is conditional on the input sequence, not a calibrated probability of improved binding. Multiple proposed mutations also need joint evaluation; adding single-mutation scores assumes away interactions between positions.

For a fair comparison, keep the labels, splits, downstream learner, and tuning budget fixed. Include a simple baseline such as amino acid composition. Fit preprocessing on training data only, repeat training with several seeds, and report both predictive quality and computing cost. If your goal is transfer to new antigens, hold out antigens as well as related VHHs.

Run both encoders without losing residue alignment

The tokenizer is an easy place to make a silent mistake. In the releases used here, nanoBERT’s RobertaTokenizer accepts a contiguous amino acid string. VHHBERT uses a BertTokenizer with space-separated residues, even though its encoder is RoBERTa. Loading the right model class does not guarantee that the input was tokenized correctly. The VHHBERT model card identifies its tokenizer class; the checks below verify every residue token.

1. Create an environment:

python3 -m venv .venv
. .venv/bin/activate
python -m pip install 'torch>=2.6' 'transformers==4.57.6'

2. Save this as compare_embeddings.py:

"""Check token alignment and embedding shapes; this is not a biological benchmark."""
import argparse
from pathlib import Path


def main():
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--sequence", default="ACDEFGHIKLMNPQRSTVWY")
    parser.add_argument("--local-root", type=Path)
    args = parser.parse_args()
    sequence = args.sequence
    if not 1 <= len(sequence) <= 179:
        parser.error("Use 1-179 residues; this demo never truncates")
    if set(sequence) - set("ACDEFGHIKLMNPQRSTVWY"):
        parser.error("Use the 20 standard uppercase amino acid letters")

    import torch
    from transformers import BertTokenizer, RobertaTokenizer, RobertaModel

    specs = [
        ("nanobert", "NaturalAntibody/nanoBERT",
         "edc8182ad89a827f8737fa572c6b5fac6197e6b0", RobertaTokenizer),
        ("vhhbert", "COGNANO/VHHBERT",
         "cd6341d2700b93cd3d8d66cef360ad01cca19a73", BertTokenizer),
    ]
    for name, repo, revision, tokenizer_class in specs:
        source = str(args.local_root / name) if args.local_root else repo
        options = {"local_files_only": True} if args.local_root else {"revision": revision}
        tokenizer = tokenizer_class.from_pretrained(source, **options)
        text = " ".join(sequence) if name == "vhhbert" else sequence
        batch = tokenizer(text, return_tensors="pt", return_special_tokens_mask=True)
        special = batch.pop("special_tokens_mask").bool()
        keep = batch["attention_mask"].bool() & ~special
        ids = batch["input_ids"][0, keep[0]].tolist()
        assert tokenizer.unk_token_id not in ids, "Unknown token in sequence"
        assert tokenizer.convert_ids_to_tokens(ids) == list(sequence), "Token mismatch"
        # Both checkpoints use a RoBERTa encoder, despite different tokenizers.
        model = RobertaModel.from_pretrained(source, add_pooling_layer=False, **options).eval()
        with torch.inference_mode():
            hidden = model(**batch).last_hidden_state
            residues = hidden[0, keep[0]]
            pooled = residues.mean(dim=0)
        assert residues.shape == (len(sequence), model.config.hidden_size)
        assert torch.isfinite(residues).all()
        print(name, "residues:", tuple(residues.shape), "pooled:", tuple(pooled.shape))
        del model


if __name__ == "__main__":
    main()

3. Check the options and run it:

python compare_embeddings.py --help
python compare_embeddings.py

Output:

nanobert residues: (20, 320) pooled: (320,)
vhhbert residues: (20, 768) pooled: (768,)

The artificial 20-residue input is a software check, not a functional nanobody. The example removes special tokens before averaging, checks exact residue-token alignment, and rejects invalid symbols or sequences longer than its conservative 179-residue limit. It never truncates silently. Use --sequence to supply a protein sequence, and verify its biological provenance separately.

The CPU check was run with Python 3.14.7, PyTorch 2.14.0, and Transformers 4.57.6 using the pinned model revisions. Both output shapes matched and all residue embeddings were finite. The first run downloads the weights. Loading the encoder without its masked-language-model head can produce an unused-head warning; this example deliberately extracts features rather than residue probabilities.

Mean pooling is only a baseline. It can dilute information concentrated in a short CDR. I would compare full-sequence pooling with region-aware features using a consistent antibody-numbering method. Different vector widths also require separately fitted downstream models; the vectors are not interchangeable.

Troubleshooting

Choose a first experiment

For a new labeled VHH dataset, start with frozen nanoBERT and VHHBERT embeddings and the same modest downstream learner. Inspect errors by sequence cluster and CDR length before increasing model complexity. Fine-tune only after establishing a useful baseline and reserving a test set that stays out of model selection.

For residue prediction, use the masked-language-model head instead of the encoder-only class in this example. The nanoBERT usage guide illustrates that interface. For reproducing the binding study, use the authors’ AVIDa-SARS-CoV-2 code, including its antigen inputs and training procedure.

The public artifacts have separate license labels: nanoBERT weights list CC BY-NC-SA 4.0, VHHBERT weights list MIT, and VHHCorpus-2M lists CC BY-NC 4.0. Keep the model and dataset records with your experiment.

For broader baselines, continue with the site’s ESM family comparison and ESM-2 embeddings tutorial.

The cover is AI-generated conceptual artwork, not an experimentally determined molecular structure.