Tip: press / anywhere to search. Or open a random note →

Handwritten notebook page explaining how AI chatbots like ChatGPT work by predicting the next token
The one-page note: “How AI Chatbots (like ChatGPT) Work”

AI & Tech Full guide See the one-page note →

How AI Chatbots Work: They Predict the Next Piece of Text

How AI chatbots work, in plain English: large language models, tokens and next-word prediction, plus why they can sound sure and still be wrong.

Good to know

General info. AI tools change fast, so check each product's own help pages for its current features.

On this page
  1. What is an AI chatbot?
  2. How do AI chatbots learn from huge amounts of text?
  3. How AI chatbots work when you ask a question
  4. Why do AI chatbots seem so smart?
  5. What to remember: it doesn't look up the truth
  6. Tips and common mistakes
  7. Related reading
  8. Sources

AI chatbots like ChatGPT work by predicting text: they write an answer one small piece at a time, each time choosing a likely next piece based on what came before. That is the whole trick, done extremely well. This guide explains how AI chatbots work in plain English, from training to the answer on your screen, and why you should still check what they tell you.

What is an AI chatbot?

An AI chatbot is a program you can type to, and it types back in normal language. Well-known examples are ChatGPT (made by OpenAI), Claude (made by Anthropic) and Gemini (made by Google).

Under the hood, each one is built on a large language model, usually shortened to LLM. An LLM is a computer model that has learned the patterns of language from a very large amount of text. The chat window is just the friendly front door to it.

How do AI chatbots learn from huge amounts of text?

Before a chatbot can talk to you, its language model goes through training. During this stage, called pretraining, the model is shown huge amounts of text and practises one job again and again: guessing what comes next.

Text is split into tokens

Computers don't read words the way we do. So the text is first chopped into small pieces called tokens. A token can be a whole word or a part of a word.

For example, a short common word might be one token, while a long or unusual word might be split into a few pieces. Everything the model reads, and everything it writes, is made of these tokens.

The model learns to predict the next token

The model sees a stretch of tokens and tries to predict the next one. It practises this over a huge amount of text, and becomes very good at predicting what comes next.

Several sources describe this the same way, including IBM's explainer, OpenAI's own research and academic papers on next-token prediction.

Training after training

The note keeps things simple, but there is one more step worth knowing. After pretraining, chatbots are usually fine-tuned so they follow instructions and behave like helpful assistants. One method is instruction tuning, which IBM describes as improving a model's ability to follow human instructions. Another is RLHF (reinforcement learning from human feedback): people rank the model's answers, and the model is trained to prefer the ones people ranked higher.

How AI chatbots work when you ask a question

When you type a message, this is roughly what happens:

  1. Your message is turned into tokens.
  2. The tokens go through the model, which works out how likely each possible next token is.
  3. One token is selected.
  4. That new token is added to the text, and the model predicts the next one.
  5. This repeats, one piece at a time, until a full answer appears.

So when you watch an answer appear word by word on screen, you are seeing something close to how it is really made. The chatbot is not pulling out a finished paragraph. It is building it one token after another.

Transformers and attention

Large language models are built on a design called the transformer, introduced in a 2017 paper by Vaswani and colleagues called "Attention Is All You Need". Its key idea is self-attention: as the model handles each token, it weighs how much every other token in the text matters to it. That helps it keep track of meaning across a whole sentence or paragraph.

The context window

There is a limit to how much text a model can consider at once. This is called the context window, and it is measured in tokens. Anything outside that window, the model can't take into account.

Why do AI chatbots seem so smart?

If a chatbot only predicts the next token, why does it seem to understand things? Because getting very good at predicting text means picking up the patterns in that text. Those patterns include grammar, facts that appear often, and steps of reasoning that people write down.

Sources describe abilities like following instructions and reasoning as side effects of that one training goal. In other words: to predict text really well, the model has to learn a lot about how language and ideas fit together.

Bigger models and more data

Researchers have found that performance tends to improve in a predictable way as models get more parameters (the model's internal settings), more data and more computing power. This is often called scaling.

For a sense of size, GPT-3 had 175 billion parameters when it was released in 2020. Newer models don't always say how big they are.

What to remember: it doesn't look up the truth

This is the most important part. By default, a chatbot does not look up the truth. It predicts what text is likely to come next. That can give a good answer, but sometimes the most likely-sounding text is simply wrong.

OpenAI's own research explains that these mistakes, often called hallucinations, come from how models are trained and how they are tested. We cover that in detail in our guide to why AI makes things up.

Note the words "by default". Many chatbots can now search the web or use other tools, which can help. But the core of how they write is still prediction.

Tips and common mistakes

  • ✓ Check important facts. Names, numbers, dates and quotes are worth confirming somewhere reliable.
  • ✓ Give clear instructions. The better your prompt, the better its prediction of what you want.
  • ✗ Trusting the tone. A chatbot can sound confident and still be wrong. Confidence comes from fluent language, not from proof.
  • ✗ Treating it like a search engine. Unless it is actually searching, it is writing from patterns it learned, not reading a live source.

Sources

How we checked this

Every fact in this article comes from the 4 sources listed above, checked against official or peer-reviewed sources where possible. AI tools help with drafting; a person checks every fact against the sources. How we research

Published and checked
Next check
Yearly (fast-moving field)

General info. AI tools change fast, so check each product's own help pages for its current features.

Spotted a mistake? Tell us and we'll fix it and log it on the corrections page.

More from AI & Tech

All AI & Tech notes →

On this page

  1. What is an AI chatbot?
  2. How do AI chatbots learn from huge amounts of text?
  3. How AI chatbots work when you ask a question
  4. Why do AI chatbots seem so smart?
  5. What to remember: it doesn't look up the truth
  6. Tips and common mistakes
  7. Related reading
  8. Sources