# 1.3 Large Language Models In Biology

Language models are systems designed to work with human language by learning patterns from lots of text. In simple terms, they try to predict what comes next in a sentence, which lets them do things like finish your sentences, answer questions, translate text, or write entire paragraphs. They are trained on large datasets and they learn grammar, meaning, and even bits of general knowledge through training. Instead of following strict rules, they rely on patterns they’ve seen before, which makes them flexible and surprisingly good at handling all kinds of language tasks.

Language models were originally developed for natural language processing, but the same principles can be applied to biological sequences such as DNA, RNA, and proteins. In this setting, sequences of amino acids or nucleotides are treated similarly to sequences of words in a sentence, allowing models to learn patterns and relationships within biological data.

Many modern AI models used in biology such as AlphaFold {cite}`nussinov2022alphafoldartificialintelligence` and ESM2 {cite}`lin2023evolutionaryscalepredictionof` are sequence-based language models. These models can also be considered foundation models, as they are trained on very large datasets and can be adapted to a wide range of downstream biological tasks. 


## 1.3.1 Transformers: The Architecture Behind Modern LLMs

Most modern natural language models, including those in biology, are built using a neural network architecture called the "transformer" {cite}`Vaswani`.

Before transformers, many sequence models relied on architectures such as recurrent neural networks (RNNs) {cite}`rumelhart1986learning` that processed sequences one element at a time. While effective, these models struggled to capture long-range relationships within sequences.

:::{note}
The  **attention** mechanism in Transformers address the limitation in capturing long-range relationships.
:::

**Attention** allows the model to examine all positions in a sequence simultaneously and determine which elements are most relevant to one another. For example, in a protein sequence, the model can learn relationships between amino acids that are far apart in the sequence but may interact in the folded structure.

Because transformers process sequences in parallel and capture long-range dependencies effectively, they have become the dominant architecture for modern language models.

```{figure} ../figures/plm_explained_2.png
:alt: plm
:align: center
:figclass: caption-centered

Figure 1.3.1: Attention mechanism in transformers allow the model to learn relationships between amino acids that are far apart. (Created with gemini-3.1-flash-image-preview)
```


## 1.3.2 Training Language Models

Language models are typically trained using a strategy known as [self-supervised learning](https://www.ibm.com/think/topics/self-supervised-learning). Training an LLM is less like "teaching a chatbot facts" and more like building a *very large*, *very flexible* pattern-recognizer, then gradually steering it toward helpful behaviors. 

The first stages of training are known as **pre-training** and **fine-tuning**. For conversational language models, an additional step called **Reinforcement Learning from Human Feedback (RLHF)** is often used. 


```{figure} ../figures/steps_training.png
:alt: training_steps
:align: center
:figclass: caption-centered

Figure 1.3.2: Training an LLM is a multi-step process: pre-training, fine-tuning, and RLHF. These steps allow large language models to first learn general patterns from massive datasets and then become more specialized and aligned with human expectations. (Created with gemini-3.1-flash-image-preview)
```

Together, pre-training, fine-tuning, and RLHF allow LLMs to first learn general patterns from massive datasets and then become more specialized and aligned with human expectations.

### Pre-training

Pre-training is the first stage in training a language model. During this stage, the model is trained on very large collections of sequences. These sequences can be natural language (i.e. sequences of words), protein sequences (i.e. sequences of amino acids), DNA sequences (i.e. sequences of nucleotides), or other sequential data.  A foundation model trained only on protein sequences is known as a Protein Language Model (PLM). 

Here, the model learns general statistical patterns present in the data. Think of these patterns as a model learning the "rules in biology". The goal is to predict the next token (next amino acid, next nucleotide etc.) given a prior sequence. One advantage of pre-training is that it does not require manually labeled data. So it is possible to train models on extremely large datasets.


As pre-training is typically performed using self-supervised learning, the model learns by solving "prediction tasks" that are automatically derived from the data itself. These prediction tasks are known as **training objectives**. The objective defines what the model is asked to predict and provides the learning signal used to update the model’s parameters.

Two common training objectives are used in language models.

**1) Next-token prediction**

Here, the model learns to predict the next token (element) in a sequence given the elements that come before it. During training, the model repeatedly observes partial sequences and attempts to predict the next token.

For example if you're working with an amino acid sequence, the model learns to predict the next amino acid. Read more on tokens [here](https://nebius.com/blog/posts/what-is-token-in-ai).

```
M A D K T L E V K → ?
```

**2) Masked token prediction**

Another common objective is *masked token prediction*, also known as masked language modeling. In this approach, some tokens in the sequence are randomly hidden, and the model must predict the missing elements using the surrounding context {cite}`devlin2019bertpretrainingof`. This objective is commonly used in autoregressive language models, such as GPT-style models.

Here the model must infer the missing amino acid in the given sequence.
```
M A D [MASK] T L E V K

Correct prediction: K
```

By training the model to predict either the next or the masked token of millions or billions of sequences, the model gradually learns "patterns" in the sequences. In natural language, this includes grammar and semantic relationships between words. In biological sequences, the model learns signals related to protein structure, evolutionary constraints, and functional motifs {cite}`gu2021domain` based on the training data. 

### Fine-tuning
After pre-training, language models can be adapted to specific tasks through a process known as fine-tuning. During fine-tuning, the pre-trained model is further trained on a smaller, task-specific dataset. Unlike pre-training, these datasets often contain labeled examples, where the correct output for each input is known {cite}`luo2022biogpt,gu2021domain`.

The purpose of this step is to teach the model to specialize in particular applications while retaining the model. For example, a protein language model that has been pre-trained on millions of protein sequences can be fine-tuned to perform tasks such as predicting protein function, identifying binding sites, or estimating the effects of mutations. Compared to pre-training, fine-tuning typically requires much less data and compute.


### Reinforcement Learning from Human Feedback (RLHF)

For many *conversational AI systems*, an additional training stage called RLHF is used to improve the quality and safety of model responses {cite}`ouyang2022training`.

In RLHF, human evaluators review multiple outputs generated by the model and rank them according to criteria such as helpfulness, accuracy, and clarity. These rankings are then used to train a reward model that scores model outputs. The language model is subsequently optimized to produce responses that receive higher reward scores.

This process helps align the model’s behavior with human expectations and improves its ability to generate responses that are useful and safe in real-world applications.

## 1.3.3 Evaluating Large Language Models

Once a language model has been trained, researchers need ways to measure its performance. This process is known as evaluation or benchmarking. Evaluation helps determine whether a model is accurate, reliable, and useful for a particular task.

For general-purpose language models, common benchmarks test abilities such as:

- Question answering
- Mathematical reasoning
- Code generation
- Reading comprehension
- Scientific knowledge

For example, benchmarks such as MMLU (Massive Multitask Language Understanding) {cite}`hendrycks2020measuring` evaluate a model across dozens of academic subjects ranging from biology and chemistry to law and history.

However, benchmark scores do not always reflect real-world performance. A model that performs well on multiple-choice questions may still struggle when solving complex scientific problems or assisting with research workflows.

More specifically, scientific applications often require more specialized evaluations. Rather than asking whether a model can answer trivia questions, researchers may want to know: Can the model design a useful experiment? Can it correctly interpret scientific literature? Can it generate hypotheses supported by evidence?

To answer these questions, researchers develop **domain-specific** benchmarks that more closely resemble real scientific tasks. For example we developed **Lab-Bench** {cite}`laurent2024lab` for this exact reason. Lab-Bench is a benchmark designed to evaluate how well language models perform scientific research in biology. Rather than testing general knowledge, Lab-Bench assesses whether models can reason about experimental procedures, laboratory protocols, and scientific workflows. A full description of the dataset is published at: [https://arxiv.org/abs/2407.10362](https://arxiv.org/abs/2407.10362). Benchmarks such as Lab-Bench provide a more realistic assessment of whether an AI system could be useful in research environments.

On the other hand, not all aspects of model performance can be measured automatically. In these cases human expert revision is used. While expert evaluations can be labor and time intensive, this approach is particularly important in biology and medicine, where incorrect information can have significant consequences.

### Limitations of benchmarking

Although benchmarks are useful, they are not perfect measures of intelligence or usefulness. Models can sometimes become optimized for specific benchmark tasks without genuinely improving their reasoning abilities. In addition, benchmark performance may not translate directly to real-world scientific applications.

For this reason, modern AI evaluation often combines:

- Automated benchmarks
- Domain-specific benchmarks
- Human expert evaluation
- Real-world testing

Together, these approaches provide a more complete picture of a model's strengths and limitations.

## 1.3.4 Examples of foundational models in biology

The table below provides a comparative overview of several major foundation models used in biological research.

```{table} Table 1.3.1: Example of foundation models in biology (Last updated: March 2026).
| Model Name | Type | Architecture | Key Application | Scale / Data |
| :--- | :--- | :--- | :--- | :--- |
| **ESM-2** {cite}`lin2023evolutionaryscalepredictionof`| Protein | Transformer Encoder (BERT-style) | Structure prediction, variant effect prediction, representation learning | Up to 15B parameters; trained on ~65M UniRef sequences  |
| **ESMFold** {cite}`lin2023evolutionaryscalepredictionof` | Protein | ESM-2 + Folding Trunk | Atomic-level protein structure prediction from single sequence (no MSA) | 3B to 15B parameters; trained on PDB + 12M AlphaFold predictions |
| **AlphaFold2** {cite}`jumper2021highly` | Protein | Evoformer + Structure Module | High-accuracy structure prediction using Multiple Sequence Alignments (MSAs) | Trained on PDB; utilizes large MSA databases (BFD, MGnify) (pqac-00000031, pqac-00000036) |
| **DNABERT / DNABERT-2** {cite}`ji2021dnabert,zhou2024dnabert`| Genomic | Transformer Encoder | Promoter identification, splice site prediction, transcription factor binding | Human genome; DNABERT-2 uses BPE tokenization and multi-species data |
| **Enformer** {cite}`avsec2021effective` | Genomic | Transformer with long-range attention  | Predicting gene expression from sequence, enhancer-promoter interactions | Long context window (100kb); trained on human/mouse genome data |
| **HyenaDNA** {cite}`nguyen2023hyenadna` | Genomic | Hyena hierarchy (Long Convolution) | Modeling long-range genomic patterns, variant classification | Up to 1M token context window; trained on human reference genome |
| **scGPT** {cite}`cui2024scgpt` | Single-Cell | Generative Transformer | Cell type annotation, perturbation prediction, batch correction | Trained on >10M single-cell profiles |
| **Geneformer** {cite}`theodoris2023transfer` | Single-Cell | Transformer | Dosage sensitivity, chromatin dynamics, gene network modeling | Trained on ~30M single-cell transcriptomes |
| **ProtGPT2** {cite}`ferruz2022protgpt2`| Protein | Transformer Decoder (GPT-style) | De novo protein sequence generation | Trained on UniRef50; 738M parameters |
| **ProGen** {cite}`madani2023large` | Protein | Transformer Decoder (Conditional) | Controllable protein sequence generation | Trained on ~280M protein sequences with taxonomic/functional tags |
| **ProteinBERT** {cite}`brandes2022proteinbert`| Protein | Transformer Encoder (BERT-style) | Protein function prediction, GO term classification | Trained on UniRef90 + Gene Ontology annotations |
```

## 1.3.5 Additional Reading

- [Hugging Face LLM Course](https://huggingface.co/learn/llm-course/chapter0/1):  *Highly recommended! A great resource for getting started on LLMs*
- [Databricks blog post on LLM pre-training](https://www.databricks.com/blog/llm-pre-training-and-custom-llms)
- [OpenAI blog post on aligning language models](https://openai.com/index/instruction-following/)
- [A Comprehensive Introduction to Fine-Tuning LLMs](https://medium.com/@sahin.samia/a-comprehensive-introduction-to-fine-tuning-llms-4d1bcc95a83a)

- [IBM blog post on fine-tuning](https://www.ibm.com/think/topics/fine-tuning)

- [Andrej Karpathy's Intro to LLMs](https://www.youtube.com/watch?v=zjkBMFhNj_g)

- [A YouTube video explaining the Transformer model](https://www.youtube.com/watch?v=wjZofJX0v4M)

- [Attention mechanism in transformers](https://www.youtube.com/watch?v=eMlx5fFNoYc)

- [7 AI Terms You Need to Know](https://www.youtube.com/watch?v=VSFuqMh4hus)

## References

```{bibliography}
:filter: docname in docnames
```