Free lesson · Module 1, lesson 4 of 6

How a language model learns: patterns from text

Explain with the phone keyboard's next-word suggestion how a model learns from a huge amount of text to predict the next piece, and why it writes fluent sentences without looking anything up.

The video for this lesson is being prepared

About 10 minutes, narrated in English, Hindi and Odia. The notes and the quiz below are open now.

Lesson notes

For the student

A language model learns by playing one game over and over: guess the next word, check the guess, adjust itself a little. That is why it writes fluent sentences, and also why a fluent sentence can be wrong.

How it learns

  • It reads some text, hides the next word, and guesses it from the words before it
  • It checks the guess against the real next word, and adjusts itself a little so the right word gets a better chance next time
  • It repeats this over a huge amount of text from many sources; nobody writes a rule for each sentence
  • To guess well it picks up grammar, common facts and how different kinds of writing sound
  • Strictly, it predicts pieces of words, which module 2 calls tokens; this lesson says the next word

What it keeps

  • Not a library: it keeps no copy of the pages it read and does not search them later
  • What changes is the model itself, a huge number of small settings (lesson 6 names them)
  • Like Meena, who has read the newspaper for years: she answers from memory in good sentences, usually right, now and then wrong about a date, with the same confidence
  • It writes an answer one next word at a time and looks nothing up while it writes, unless an app adds a web search

The worked examples

  • Nisha types "Reached the railway" and the keyboard suggests "station": in the text it learned from, station usually came next. No rule, no search
  • Six made-up messages: after Happy came birthday 3 times, Diwali 2, journey 1. The toy model predicts birthday. After the first message alone it could only guess journey: more text, a better guess. This toy is only a picture of the idea. A real model keeps no count table and does not store the messages: after each guess it adjusts its settings a little, over a huge amount of text, and it looks at many earlier words
  • An assistant with no web search said a village library opened in 1965; the plaque says 1982. After "opened in" a year usually comes next, so it wrote a likely-looking year. Predicted, not looked up

The habit to keep

  • A reply is likely words, not a looked-up record
  • Ask of each sentence: is this a pattern, or a fact someone must have recorded?
  • Wording, tone, a first draft: the pattern is what you want
  • Names, dates, numbers, amounts: check a record (an official notice, a textbook, a receipt)
  • The smaller and more local the fact, the more checking it needs

The trap

  • Wrong: it looks up each answer; it keeps every page and finds the right one; a fluent, confident reply must be correct
  • Right: it predicts the next word from learned patterns; it keeps patterns, not pages; fluent shows a common pattern, not a checked fact; check names, dates and numbers

Try it

  • On your phone, type one word in a message, then tap the keyboard's top suggestion ten times. Read the sentence it built: each word is a likely next word, and nothing in it was checked
  • Ask a chat assistant a small local fact you can verify, such as the year your school or a nearby temple was founded
  • Compare its answer with a record: the plaque, the school office, or someone who knows. Was the reply fluent? Was it right? If the assistant shows that it searched the web, note that too
  • Write both answers in your notes file

Lesson quiz

Checked in your browser. Nothing is sent or saved.

5 questions

Question 1 · easy

While a language model is learning, what does it do again and again with text?

Question 2 · easy

Arjun types into a chat assistant: "Explain in simple words how to cancel a train ticket", and pastes the SMS he got when he booked. In AI words, what is the text he sends?

Question 3 · medium

Say a toy model has read only these seven made-up messages, in this order:
- Please wait here.
- Please call me tonight.
- Please call Meena.
- Please send the bill.
- Please call back.
- Please call after lunch.
- Please send the photo.

Counting, which word does it predict right after Please?

Question 4 · medium

Nisha asks a chat assistant to draft a letter to her municipal office about a broken streetlight. The draft has a polite opening, her ward number, the date she first complained, and the office's helpline number. Which parts must she check against a record before sending?

Question 5 · hard

A made-up announcement makes four claims about a new model:
1: Anyone can download it and run it, under its licence.
2: It learned from a large collection of Odia news and textbooks.
3: It replies to a long question in about 3 seconds on a laptop.
4: It has 12 billion parameters.

Which tagging of the claims is right?

Your first conversation with a chat assistantNext: What LLMs do well and badly