1.3 Large Language Models In Biology

Language models are systems designed to work with human language by learning patterns from lots of text. In simple terms, they try to predict what comes next in a sentence, which lets them do things like finish your sentences, answer questions, translate text, or write entire paragraphs. They are trained on large datasets and they learn grammar, meaning, and even bits of general knowledge through training. Instead of following strict rules, they rely on patterns they’ve seen before, which makes them flexible and surprisingly good at handling all kinds of language tasks.

Language models were originally developed for natural language processing, but the same principles can be applied to biological sequences such as DNA, RNA, and proteins. In this setting, sequences of amino acids or nucleotides are treated similarly to sequences of words in a sentence, allowing models to learn patterns and relationships within biological data.

Many modern AI models used in biology such as AlphaFold [NZLJ22] and ESM2 [LAR+23] are sequence-based language models. These models can also be considered foundation models, as they are trained on very large datasets and can be adapted to a wide range of downstream biological tasks.

1.3.1 Transformers: The Architecture Behind Modern LLMs

Most modern natural language models, including those in biology, are built using a neural network architecture called the “transformer” [VSP+17].

Before transformers, many sequence models relied on architectures such as recurrent neural networks (RNNs) [RHW86] that processed sequences one element at a time. While effective, these models struggled to capture long-range relationships within sequences.

Note

The attention mechanism in Transformers address the limitation in capturing long-range relationships.

Attention allows the model to examine all positions in a sequence simultaneously and determine which elements are most relevant to one another. For example, in a protein sequence, the model can learn relationships between amino acids that are far apart in the sequence but may interact in the folded structure.

Because transformers process sequences in parallel and capture long-range dependencies effectively, they have become the dominant architecture for modern language models.

Fig. 4 Figure 1.3.1: Attention mechanism in transformers allow the model to learn relationships between amino acids that are far apart. (Created with gemini-3.1-flash-image-preview)

1.3.2 Training Language Models

Language models are typically trained using a strategy known as self-supervised learning. Training an LLM is less like “teaching a chatbot facts” and more like building a very large, very flexible pattern-recognizer, then gradually steering it toward helpful behaviors.

The first stages of training are known as pre-training and fine-tuning. For conversational language models, an additional step called Reinforcement Learning from Human Feedback (RLHF) is often used.

Fig. 5 Figure 1.3.2: Training an LLM is a multi-step process: pre-training, fine-tuning, and RLHF. These steps allow large language models to first learn general patterns from massive datasets and then become more specialized and aligned with human expectations. (Created with gemini-3.1-flash-image-preview)

Together, pre-training, fine-tuning, and RLHF allow LLMs to first learn general patterns from massive datasets and then become more specialized and aligned with human expectations.

Pre-training

Pre-training is the first stage in training a language model. During this stage, the model is trained on very large collections of sequences. These sequences can be natural language (i.e. sequences of words), protein sequences (i.e. sequences of amino acids), DNA sequences (i.e. sequences of nucleotides), or other sequential data. A foundation model trained only on protein sequences is known as a Protein Language Model (PLM).

Here, the model learns general statistical patterns present in the data. Think of these patterns as a model learning the “rules in biology”. The goal is to predict the next token (next amino acid, next nucleotide etc.) given a prior sequence. One advantage of pre-training is that it does not require manually labeled data. So it is possible to train models on extremely large datasets.

As pre-training is typically performed using self-supervised learning, the model learns by solving “prediction tasks” that are automatically derived from the data itself. These prediction tasks are known as training objectives. The objective defines what the model is asked to predict and provides the learning signal used to update the model’s parameters.

Two common training objectives are used in language models.

1) Next-token prediction

Here, the model learns to predict the next token (element) in a sequence given the elements that come before it. During training, the model repeatedly observes partial sequences and attempts to predict the next token.

For example if you’re working with an amino acid sequence, the model learns to predict the next amino acid. Read more on tokens here.

M A D K T L E V K → ?

2) Masked token prediction

Another common objective is masked token prediction, also known as masked language modeling. In this approach, some tokens in the sequence are randomly hidden, and the model must predict the missing elements using the surrounding context [DCLT19]. This objective is commonly used in autoregressive language models, such as GPT-style models.

Here the model must infer the missing amino acid in the given sequence.

M A D [MASK] T L E V K

Correct prediction: K

By training the model to predict either the next or the masked token of millions or billions of sequences, the model gradually learns “patterns” in the sequences. In natural language, this includes grammar and semantic relationships between words. In biological sequences, the model learns signals related to protein structure, evolutionary constraints, and functional motifs [GTC+21] based on the training data.

Fine-tuning

After pre-training, language models can be adapted to specific tasks through a process known as fine-tuning. During fine-tuning, the pre-trained model is further trained on a smaller, task-specific dataset. Unlike pre-training, these datasets often contain labeled examples, where the correct output for each input is known [GTC+21, LSX+22].

The purpose of this step is to teach the model to specialize in particular applications while retaining the model. For example, a protein language model that has been pre-trained on millions of protein sequences can be fine-tuned to perform tasks such as predicting protein function, identifying binding sites, or estimating the effects of mutations. Compared to pre-training, fine-tuning typically requires much less data and compute.

Reinforcement Learning from Human Feedback (RLHF)

For many conversational AI systems, an additional training stage called RLHF is used to improve the quality and safety of model responses [OWJ+22].

In RLHF, human evaluators review multiple outputs generated by the model and rank them according to criteria such as helpfulness, accuracy, and clarity. These rankings are then used to train a reward model that scores model outputs. The language model is subsequently optimized to produce responses that receive higher reward scores.

This process helps align the model’s behavior with human expectations and improves its ability to generate responses that are useful and safe in real-world applications.

1.3.3 Evaluating Large Language Models

Once a language model has been trained, researchers need ways to measure its performance. This process is known as evaluation or benchmarking. Evaluation helps determine whether a model is accurate, reliable, and useful for a particular task.

For general-purpose language models, common benchmarks test abilities such as:

• Question answering

• Mathematical reasoning

• Code generation

• Reading comprehension

• Scientific knowledge

For example, benchmarks such as MMLU (Massive Multitask Language Understanding) [HBB+20] evaluate a model across dozens of academic subjects ranging from biology and chemistry to law and history.

However, benchmark scores do not always reflect real-world performance. A model that performs well on multiple-choice questions may still struggle when solving complex scientific problems or assisting with research workflows.

More specifically, scientific applications often require more specialized evaluations. Rather than asking whether a model can answer trivia questions, researchers may want to know: Can the model design a useful experiment? Can it correctly interpret scientific literature? Can it generate hypotheses supported by evidence?

To answer these questions, researchers develop domain-specific benchmarks that more closely resemble real scientific tasks. For example we developed Lab-Bench [LJR+24] for this exact reason. Lab-Bench is a benchmark designed to evaluate how well language models perform scientific research in biology. Rather than testing general knowledge, Lab-Bench assesses whether models can reason about experimental procedures, laboratory protocols, and scientific workflows. A full description of the dataset is published at: https://arxiv.org/abs/2407.10362. Benchmarks such as Lab-Bench provide a more realistic assessment of whether an AI system could be useful in research environments.

On the other hand, not all aspects of model performance can be measured automatically. In these cases human expert revision is used. While expert evaluations can be labor and time intensive, this approach is particularly important in biology and medicine, where incorrect information can have significant consequences.

Limitations of benchmarking

Although benchmarks are useful, they are not perfect measures of intelligence or usefulness. Models can sometimes become optimized for specific benchmark tasks without genuinely improving their reasoning abilities. In addition, benchmark performance may not translate directly to real-world scientific applications.

For this reason, modern AI evaluation often combines:

• Automated benchmarks

• Domain-specific benchmarks

• Human expert evaluation

• Real-world testing

Together, these approaches provide a more complete picture of a model’s strengths and limitations.

1.3.4 Examples of foundational models in biology

The table below provides a comparative overview of several major foundation models used in biological research.

Table 2 Table 1.3.1: Example of foundation models in biology (Last updated: March 2026).

Model Name	Type	Architecture	Key Application	Scale / Data
ESM-2 [LAR+23]	Protein	Transformer Encoder (BERT-style)	Structure prediction, variant effect prediction, representation learning	Up to 15B parameters; trained on ~65M UniRef sequences
ESMFold [LAR+23]	Protein	ESM-2 + Folding Trunk	Atomic-level protein structure prediction from single sequence (no MSA)	3B to 15B parameters; trained on PDB + 12M AlphaFold predictions
AlphaFold2 [JEP+21]	Protein	Evoformer + Structure Module	High-accuracy structure prediction using Multiple Sequence Alignments (MSAs)	Trained on PDB; utilizes large MSA databases (BFD, MGnify) (pqac-00000031, pqac-00000036)
DNABERT / DNABERT-2 [JZLD21, ZJL+24]	Genomic	Transformer Encoder	Promoter identification, splice site prediction, transcription factor binding	Human genome; DNABERT-2 uses BPE tokenization and multi-species data
Enformer [AAV+21]	Genomic	Transformer with long-range attention	Predicting gene expression from sequence, enhancer-promoter interactions	Long context window (100kb); trained on human/mouse genome data
HyenaDNA [NPF+23]	Genomic	Hyena hierarchy (Long Convolution)	Modeling long-range genomic patterns, variant classification	Up to 1M token context window; trained on human reference genome
scGPT [CWM+24]	Single-Cell	Generative Transformer	Cell type annotation, perturbation prediction, batch correction	Trained on >10M single-cell profiles
Geneformer [TXC+23]	Single-Cell	Transformer	Dosage sensitivity, chromatin dynamics, gene network modeling	Trained on ~30M single-cell transcriptomes
ProtGPT2 [FSHocker22]	Protein	Transformer Decoder (GPT-style)	De novo protein sequence generation	Trained on UniRef50; 738M parameters
ProGen [MKG+23]	Protein	Transformer Decoder (Conditional)	Controllable protein sequence generation	Trained on ~280M protein sequences with taxonomic/functional tags
ProteinBERT [BOP+22]	Protein	Transformer Encoder (BERT-style)	Protein function prediction, GO term classification	Trained on UniRef90 + Gene Ontology annotations

1.3.5 Additional Reading

• Hugging Face LLM Course: Highly recommended! A great resource for getting started on LLMs

• Databricks blog post on LLM pre-training

• OpenAI blog post on aligning language models

• A Comprehensive Introduction to Fine-Tuning LLMs

• IBM blog post on fine-tuning

• Andrej Karpathy’s Intro to LLMs

• A YouTube video explaining the Transformer model

• Attention mechanism in transformers

• 7 AI Terms You Need to Know

References

[AAV+21]

Žiga Avsec, Vikram Agarwal, Daniel Visentin, Joseph R Ledsam, Agnieszka Grabska-Barwinska, Kyle R Taylor, Yannis Assael, John Jumper, Pushmeet Kohli, and David R Kelley. Effective gene expression prediction from sequence by integrating long-range interactions. Nature methods, 18(10):1196–1203, 2021.

[BOP+22]

Nadav Brandes, Dan Ofer, Yam Peleg, Nadav Rappoport, and Michal Linial. Proteinbert: a universal deep-learning model of protein sequence and function. Bioinformatics, 38(8):2102–2110, 2022.

[CWM+24]

Haotian Cui, Chloe Wang, Hassaan Maan, Kuan Pang, Fengning Luo, Nan Duan, and Bo Wang. Scgpt: toward building a foundation model for single-cell multi-omics using generative ai. Nature methods, 21(8):1470–1480, 2024.

[DCLT19]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: pre-training of deep bidirectional transformers for language understanding. ArXiv, pages 4171–4186, 2019. URL: https://doi.org/10.48550/arxiv.1810.04805, doi:10.48550/arxiv.1810.04805.

[FSHocker22]

Noelia Ferruz, Steffen Schmidt, and Birte Höcker. Protgpt2 is a deep unsupervised language model for protein design. Nature communications, 13(1):4348, 2022.

[GTC+21] (1,2)

Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare, 3:1–23, Oct 2021. URL: https://doi.org/10.1145/3458754, doi:10.1145/3458754.

[HBB+20]

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.

[JZLD21]

Yanrong Ji, Zhihan Zhou, Han Liu, and Ramana V Davuluri. Dnabert: pre-trained bidirectional encoder representations from transformers model for dna-language in genome. Bioinformatics, 37(15):2112–2120, 2021.

[JEP+21]

John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, and others. Highly accurate protein structure prediction with alphafold. nature, 596(7873):583–589, 2021.

[LJR+24]

Jon M Laurent, Joseph D Janizek, Michael Ruzo, Michaela M Hinks, Michael J Hammerling, Siddharth Narayanan, Manvitha Ponnapati, Andrew D White, and Samuel G Rodriques. Lab-bench: measuring capabilities of language models for biology research. arXiv preprint arXiv:2407.10362, 2024.

[LAR+23] (1,2,3)

Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, Allan dos Santos Costa, Maryam Fazel-Zarandi, Tom Sercu, Salvatore Candido, and Alexander Rives. Evolutionary-scale prediction of atomic-level protein structure with a language model. Mar 2023. URL: https://doi.org/10.1126/science.ade2574, doi:10.1126/science.ade2574.

[LSX+22]

Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tie-Yan Liu. Biogpt: generative pre-trained transformer for biomedical text generation and mining. Briefings in bioinformatics, Sep 2022. URL: https://doi.org/10.48550/arxiv.2210.10341, doi:10.48550/arxiv.2210.10341.

[MKG+23]

Ali Madani, Ben Krause, Eric R Greene, Subu Subramanian, Benjamin P Mohr, James M Holton, Jose Luis Olmos Jr, Caiming Xiong, Zachary Z Sun, Richard Socher, and others. Large language models generate functional protein sequences across diverse families. Nature biotechnology, 41(8):1099–1106, 2023.

[NPF+23]

Eric Nguyen, Michael Poli, Marjan Faizi, Armin Thomas, Michael Wornow, Callum Birch-Sykes, Stefano Massaroli, Aman Patel, Clayton Rabideau, Yoshua Bengio, and others. Hyenadna: long-range genomic sequence modeling at single nucleotide resolution. Advances in neural information processing systems, 36:43177–43201, 2023.

[NZLJ22]

Ruth Nussinov, Mingzhen Zhang, Yonglan Liu, and Hyunbum Jang. Alphafold, artificial intelligence (ai), and allostery. The Journal of Physical Chemistry. B, 126:6372–6383, Aug 2022. URL: https://doi.org/10.1021/acs.jpcb.2c04346, doi:10.1021/acs.jpcb.2c04346.

[OWJ+22]

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and others. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022.

[RHW86]

David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back-propagating errors. nature, 323(6088):533–536, 1986.

[TXC+23]

Christina V Theodoris, Ling Xiao, Anant Chopra, Mark D Chaffin, Zeina R Al Sayed, Matthew C Hill, Helene Mantineo, Elizabeth M Brydon, Zexian Zeng, X Shirley Liu, and others. Transfer learning enables predictions in network biology. Nature, 618(7965):616–624, 2023.

[VSP+17]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL: https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf.

[ZJL+24]

Zhihan Zhou, Yanrong Ji, Weijian Li, Pratik Dutta, Ramana Davuluri, and Han Liu. Dnabert-2: efficient foundation model and benchmark for multi-species genomes. In International Conference on Learning Representations, volume 2024, 41642–41665. 2024.