The New Age of Software: Why GitHub Is Now at the Heart of the AI Coding Revolution

AI-generated, human-reviewed.

A large language model generates answers by predicting the next piece of text, using billions of numbers shaped by training. That explains how a chatbot can answer a question with the Wi-Fi off, but it also leaves room for mistakes that sound convincing.

Quick Summary

On Hands-On AI, Mikah Sargent returned to the Qwen3.5 9B model that answered his torque wrench question offline in Episode 1. This time, he examined what was in the file and how it produced those answers.

Running a chatbot on your own computer makes the process easier to see. The same basics also help explain the chatbots you use through an app or website.

What's in the Model File?

Using Ollama, a free app for running models locally, Mikah checked the model with ollama show qwen3.5:9b. His copy contained about 9 billion parameters in a language-model file of roughly 5.6 GB.

Parameters are numbers whose values were learned during training. Mikah describes them as knobs: training adjusts them until the model gets better at predicting text.

At 16-bit precision, 9 billion numbers would occupy about 18 GB. Quantization reduces the precision used to store them, making the download smaller at some cost to answer quality. The Q4_K_M version Mikah used averages about five bits per parameter once the extra information needed to decode the numbers is included.

How Your Words Become Tokens

Before the model processes your prompt, it breaks the text into tokens. A token might be a whole word, part of a word, or a punctuation mark, and different model families divide text differently.

In English, 100 tokens is roughly 75 words. That's a useful estimate when an app lists a token limit or tells you how fast a model is generating text.

It also helps explain why counting letters has been surprisingly difficult for chatbots. In Episode 1, Qwen counted the R's in “strawberry” by first spelling out the word, working around the fact that it normally processes chunks of text.

How the Model Chooses What Comes Next

The model uses your prompt and everything it has written so far to score the possible next tokens. It selects one, adds it to the response, and repeats until the answer is finished.

Those scores describe how likely a piece of text is to come next. They don't measure whether the answer is true.

The temperature setting affects that selection. Higher temperatures give lower-ranked choices more of a chance; at zero, the model selects the highest-scoring token. Lowering it can make responses more repeatable, but it doesn't check the facts.

How Training Turns Numbers Into an Assistant

During pretraining, a model repeatedly predicts text, compares its predictions with the training material, and adjusts its parameters. Post-training can then use example conversations and feedback to make it better at responding as an assistant.

Once training ends, chatting with this downloaded model doesn't change its parameters. Your conversation gives it context for the next response, while a cloud chatbot's memory feature is a separate layer.

Why a Wrong Answer Can Sound Right

In this local setup, generating an answer doesn't include an automatic check against a reliable source. The model can produce a plausible sentence even when it lacks the information needed to answer accurately.

Mikah explored that problem by asking about the first episode of Hands-On AI with the Wi-Fi off. The episode came out after the model's release, and a fresh chat gave it neither information about the show nor a way to look it up.

How to Apply This

• Give it something to work from. Paste the email, document, or other source you want it to summarize or explain.
• Check recent and obscure facts. These are especially good reasons to consult a source rather than rely on the model's answer alone.
• Try the tools yourself. Run ollama show on a downloaded model, or put your name into a tokenizer to see how it splits.

Key Takeaways

• Parameters hold patterns learned during training.
• Tokens are the pieces of text a model processes.
• Temperature affects which tokens get selected, not factual accuracy.

The Bottom Line

A chatbot is useful when you give it a clear task and relevant material. Understanding how it produces an answer gives you a better sense of what to check, especially when a mistake would matter.

Watch the full episode and subscribe: https://twit.tv/shows/hands-on-ai/episodes/2