{"cells":[{"cell_type":"markdown","metadata":{"id":"xTm6yfBrV0od"},"source":["# From Words to Vectors: Understanding the Heart of LLMs\n","\n","## Introduction\n","\n","Large Language Models (LLMs) like GPT and Claude can write essays, answer questions, and even generate code. But how does a model that only understands *numbers* deal with *words*?\n","\n","The answer lies in a deceptively simple idea: **turn every word into a list of numbers** — a *vector*.\n","\n","This notebook will take you step by step through:\n","1. Why computers need numbers, not words\n","2. The simplest encoding — **one-hot vectors**\n","3. Why one-hot is not enough\n","4. **Word embeddings**: vectors that capture meaning\n","5. Exploring geometry in embedding space\n","\n","No GPU needed. Everything runs on your laptop in seconds."],"id":"xTm6yfBrV0od"},{"cell_type":"code","execution_count":null,"metadata":{"id":"sQm7axVmV0og"},"outputs":[],"source":["!pip install gensim\n","import numpy as np\n","import matplotlib.pyplot as plt\n","from sklearn.decomposition import PCA\n","\n","# We'll use gensim's pre-built small word vectors (no training required)\n","# Install once: pip install gensim\n","import gensim.downloader as api\n","\n","print(\"All imports OK!\")"],"id":"sQm7axVmV0og"},{"cell_type":"markdown","metadata":{"id":"cgpMReFWV0oh"},"source":["---\n","## Part 1 — The Problem: Computers Don't Understand Words\n","\n","A machine learning model is, at its core, a big mathematical function. It multiplies, adds, and applies non-linearities to numbers. It has **no built-in concept of 'dog', 'cat', or 'run'**.\n","\n","So the first challenge is: how do we convert a word like `\"king\"` into something a model can compute with?\n","\n","The simplest idea: build a **vocabulary** of all words we care about, and represent each word as its position in that vocabulary."],"id":"cgpMReFWV0oh"},{"cell_type":"code","execution_count":null,"metadata":{"id":"7jvtRuzRV0oh"},"outputs":[],"source":["# Our toy vocabulary\n","vocabulary = [\"king\", \"queen\", \"man\", \"woman\", \"cat\", \"dog\", \"run\", \"sleep\"]\n","word_to_index = {word: i for i, word in enumerate(vocabulary)}\n","\n","print(\"Vocabulary:\", vocabulary)\n","print(\"\\nIndex lookup:\")\n","for word, idx in word_to_index.items():\n","    print(f\"  '{word}' → index {idx}\")"],"id":"7jvtRuzRV0oh"},{"cell_type":"markdown","metadata":{"id":"0SAZqXdYV0oi"},"source":["Good — each word now has a unique integer. But integers carry misleading information: is `\"king\"` (index 0) somehow *closer* to `\"queen\"` (index 1) than to `\"cat\"` (index 4)? Not necessarily. We need something better.\n","\n","### One-Hot Encoding\n","\n","The classic fix: represent each word as a vector of **all zeros, except a single 1** at the word's index position."],"id":"0SAZqXdYV0oi"},{"cell_type":"code","execution_count":null,"metadata":{"id":"Ez1j_gobV0oi"},"outputs":[],"source":["def one_hot(word, vocab):\n","    \"\"\"Return a one-hot vector for `word` given `vocab` list.\"\"\"\n","    vec = np.zeros(len(vocab))\n","    vec[word_to_index[word]] = 1.0\n","    return vec\n","\n","# Let's see what 'king' and 'queen' look like\n","v_king  = one_hot(\"king\",  vocabulary)\n","v_queen = one_hot(\"queen\", vocabulary)\n","v_cat   = one_hot(\"cat\",   vocabulary)\n","\n","print(\"one-hot('king') :\", v_king)\n","print(\"one-hot('queen'):\", v_queen)\n","print(\"one-hot('cat')  :\", v_cat)"],"id":"Ez1j_gobV0oi"},{"cell_type":"markdown","metadata":{"id":"EBdfYA7wV0oi"},"source":["### ❓ Question 1\n","\n","> **Look at the one-hot vectors for `\"king\"` and `\"queen\"`. Now compute the dot product (similarity) between them and between `\"king\"` and `\"cat\"`. What do you get? What does this tell you about one-hot encoding's ability to capture meaning?**\n",">\n","> *Hint: use `np.dot(v_king, v_queen)` and `np.dot(v_king, v_cat)`.*"],"id":"EBdfYA7wV0oi"},{"cell_type":"code","execution_count":null,"metadata":{"id":"jyge5D0cV0oi"},"outputs":[],"source":["# Your exploration here:\n","sim_king_queen = np.dot(v_king, v_queen)\n","sim_king_cat   = np.dot(v_king, v_cat)\n","\n","print(f\"dot(king, queen) = {sim_king_queen}\")\n","print(f\"dot(king, cat)   = {sim_king_cat}\")\n","\n","# What do you notice? Write your answer as a comment below:\n","# YOUR ANSWER: ..."],"id":"jyge5D0cV0oi"},{"cell_type":"markdown","metadata":{"id":"W-GKB0q7V0oj"},"source":["---\n","## Part 2 — The Problem with One-Hot Vectors\n","\n","One-hot vectors have a critical flaw: **every word is equally far from every other word**. The dot product between any two different one-hot vectors is always 0.\n","\n","This means the model can never learn that `\"king\"` and `\"queen\"` are more related to each other than either is to `\"run\"`.\n","\n","Also, in real applications vocabularies have **50,000+ words**. Each one-hot vector would be 50,000 dimensions, almost entirely zeros — very wasteful.\n","\n","We need vectors that are:\n","- **Dense** (all dimensions carry information, not mostly zeros)\n","- **Meaningful** (similar words have similar vectors)\n","\n","This is exactly what **word embeddings** provide."],"id":"W-GKB0q7V0oj"},{"cell_type":"markdown","metadata":{"id":"tL9hxNAqV0oj"},"source":["---\n","## Part 3 — Word Embeddings: Learning Meaning from Context\n","\n","The big idea (Word2Vec, 2013): **\"a word is defined by the company it keeps.\"**\n","\n","If `\"king\"` and `\"queen\"` tend to appear near the same words (`throne`, `crown`, `palace`), they should have similar vectors.\n","\n","We train a small neural network on a huge text corpus. The network learns to *predict* surrounding words from a center word. The weight matrix it learns — one row per word — becomes our embedding table.\n","\n","We won't train from scratch (that needs gigabytes of text and hours of compute). Instead, we load **pre-trained GloVe vectors** — small, 50-dimensional, trained on Wikipedia."],"id":"tL9hxNAqV0oj"},{"cell_type":"code","execution_count":null,"metadata":{"id":"CAq8skgmV0oj"},"outputs":[],"source":["# Download a compact pre-trained model (~66 MB, may take ~1 min first time)\n","# 'glove-wiki-gigaword-50' = 400k words, each a 50-dimensional vector\n","print(\"Loading pre-trained GloVe vectors (50 dimensions)...\")\n","glove = api.load('glove-wiki-gigaword-50')\n","print(f\"Vocabulary size: {len(glove):,}\")\n","print(f\"Each word vector has {glove.vector_size} dimensions\")"],"id":"CAq8skgmV0oj"},{"cell_type":"code","execution_count":null,"metadata":{"id":"lu5vVXxDV0oj"},"outputs":[],"source":["# Let's look at the actual vector for 'king'\n","king_vec = glove['king']\n","print(\"Vector for 'king':\")\n","print(king_vec)\n","print(f\"\\nShape: {king_vec.shape}  (50 numbers)\")"],"id":"lu5vVXxDV0oj"},{"cell_type":"markdown","metadata":{"id":"HqQ4rTAJV0oj"},"source":["Unlike a one-hot vector (one `1`, rest `0`), the embedding vector has **50 meaningful numbers** — each dimension encodes some latent feature of the word's usage and meaning."],"id":"HqQ4rTAJV0oj"},{"cell_type":"markdown","metadata":{"id":"wMj-WZvyV0oj"},"source":["---\n","## Part 4 — Cosine Similarity: Measuring How Close Two Words Are\n","\n","To compare two vectors, we use **cosine similarity**: the cosine of the angle between them.\n","\n","$$\\text{cosine\\_similarity}(\\mathbf{a}, \\mathbf{b}) = \\frac{\\mathbf{a} \\cdot \\mathbf{b}}{\\|\\mathbf{a}\\|\\,\\|\\mathbf{b}\\|}$$\n","\n","- Result = **1.0** → vectors point in the same direction (very similar)\n","- Result = **0.0** → perpendicular (unrelated)\n","- Result = **-1.0** → opposite directions (opposite meaning)\n","\n","We normalize by the lengths so that vector magnitude doesn't dominate — only *direction* matters."],"id":"wMj-WZvyV0oj"},{"cell_type":"code","execution_count":null,"metadata":{"id":"7T2wlpbQV0oj"},"outputs":[],"source":["def cosine_similarity(a, b):\n","    \"\"\"Cosine similarity between two numpy vectors.\"\"\"\n","    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))\n","\n","# Compare some word pairs\n","pairs = [\n","    (\"king\",  \"queen\"),\n","    (\"king\",  \"man\"),\n","    (\"king\",  \"banana\"),\n","    (\"dog\",   \"cat\"),\n","    (\"dog\",   \"automobile\"),\n","    (\"happy\", \"joyful\"),\n","    (\"happy\", \"sad\"),\n","]\n","\n","print(f\"{'Word A':<10} {'Word B':<12} {'Cosine Similarity':>18}\")\n","print(\"-\" * 42)\n","for a, b in pairs:\n","    sim = cosine_similarity(glove[a], glove[b])\n","    print(f\"{a:<10} {b:<12} {sim:>18.4f}\")"],"id":"7T2wlpbQV0oj"},{"cell_type":"markdown","metadata":{"id":"IMcwoNF8V0oj"},"source":["### ❓ Question 2\n","\n","> **Look at the similarities above. Does the ordering match your intuitions about word meaning? Pick one pair that surprises you (either higher or lower than you expected) and write down why you think that happened.**\n",">\n","> *Hint: remember that the vectors were trained on Wikipedia text — their similarity reflects how the words are used in that corpus, not a dictionary definition.*"],"id":"IMcwoNF8V0oj"},{"cell_type":"code","execution_count":null,"metadata":{"id":"TeCB9l3HV0ok"},"outputs":[],"source":["# Try your own word pairs!\n","my_word_a = \"coffee\"\n","my_word_b = \"tea\"\n","\n","sim = cosine_similarity(glove[my_word_a], glove[my_word_b])\n","print(f\"similarity('{my_word_a}', '{my_word_b}') = {sim:.4f}\")\n","\n","# Change the words above and experiment.\n","# YOUR OBSERVATION: ..."],"id":"TeCB9l3HV0ok"},{"cell_type":"markdown","metadata":{"id":"sQPd70pwV0ok"},"source":["---\n","## Part 5 — The Famous Analogy: king − man + woman ≈ queen\n","\n","Here is one of the most celebrated properties of word embeddings:\n","\n","> **Vector arithmetic on word meanings actually works.**\n","\n","The idea: if we take the vector for `\"king\"`, subtract the concept of `\"man\"`, and add the concept of `\"woman\"`, we should land close to `\"queen\"`.\n","\n","$$\\vec{\\text{king}} - \\vec{\\text{man}} + \\vec{\\text{woman}} \\approx \\vec{\\text{queen}}$$\n","\n","This tells us that the embedding space has encoded a **gender direction** — a meaningful axis."],"id":"sQPd70pwV0ok"},{"cell_type":"code","execution_count":null,"metadata":{"id":"4e3Zd8vmV0ok"},"outputs":[],"source":["# Compute the analogy vector\n","analogy_vec = glove['king'] - glove['man'] + glove['woman']\n","\n","# Find the closest words to that result vector\n","most_similar = glove.similar_by_vector(analogy_vec, topn=5)\n","\n","print(\"king − man + woman  →  closest words:\")\n","print()\n","for word, score in most_similar:\n","    print(f\"  {word:<15} (cosine similarity: {score:.4f})\")"],"id":"4e3Zd8vmV0ok"},{"cell_type":"code","execution_count":null,"metadata":{"id":"pn7WQCJmV0ok"},"outputs":[],"source":["# Try another analogy: Paris is to France as Berlin is to ?\n","# paris - france + germany ≈ berlin ?\n","\n","analogy_vec2 = glove['paris'] - glove['france'] + glove['germany']\n","most_similar2 = glove.similar_by_vector(analogy_vec2, topn=5)\n","\n","print(\"paris − france + germany  →  closest words:\")\n","print()\n","for word, score in most_similar2:\n","    print(f\"  {word:<15} (cosine similarity: {score:.4f})\")"],"id":"pn7WQCJmV0ok"},{"cell_type":"markdown","metadata":{"id":"LhF2HQD2V0ok"},"source":["### ❓ Question 3\n","\n","> **Why does vector arithmetic on word embeddings encode semantic relationships like gender or country–capital pairs? In your own words, explain the intuition. What does it say about how the embedding space is organised?**\n",">\n","> *Hint: think about what it means for the difference vector `king − man` to be similar to the difference vector `queen − woman`.*"],"id":"LhF2HQD2V0ok"},{"cell_type":"code","execution_count":null,"metadata":{"id":"nvTTnmS2V0ok"},"outputs":[],"source":["# Explore your own analogy!\n","# Template: A - B + C ≈ ?\n","A, B, C = \"actor\", \"man\", \"woman\"  # change these!\n","\n","result_vec = glove[A] - glove[B] + glove[C]\n","results = glove.similar_by_vector(result_vec, topn=5)\n","\n","print(f\"{A} − {B} + {C}  →  closest words:\")\n","for word, score in results:\n","    print(f\"  {word:<15} {score:.4f}\")"],"id":"nvTTnmS2V0ok"},{"cell_type":"markdown","metadata":{"id":"ZwggDL_TV0ok"},"source":["---\n","## Part 6 — Visualising the Embedding Space\n","\n","Our vectors live in 50 dimensions — impossible to plot directly. We use **PCA (Principal Component Analysis)** to project them down to 2D while preserving as much structure as possible.\n","\n","Think of it as finding the two most informative \"views\" of the high-dimensional cloud of points."],"id":"ZwggDL_TV0ok"},{"cell_type":"code","execution_count":null,"metadata":{"id":"bSjb19rZV0ok"},"outputs":[],"source":["# Words we want to visualise\n","words = [\n","    \"king\", \"queen\", \"prince\", \"princess\",\n","    \"man\",  \"woman\", \"boy\",    \"girl\",\n","    \"dog\",  \"cat\",   \"fish\",   \"bird\",\n","    \"paris\", \"london\", \"berlin\", \"rome\",\n","    \"france\", \"england\", \"germany\", \"italy\",\n","]\n","\n","# Collect their 50-d vectors\n","vectors = np.array([glove[w] for w in words])\n","\n","# Project to 2D with PCA\n","pca = PCA(n_components=2)\n","coords = pca.fit_transform(vectors)\n","\n","# Plot\n","fig, ax = plt.subplots(figsize=(12, 8))\n","\n","# Colour groups for readability\n","groups = {\n","    \"royalty\":   ([\"king\",\"queen\",\"prince\",\"princess\"], \"royalblue\"),\n","    \"people\":    ([\"man\",\"woman\",\"boy\",\"girl\"],         \"tomato\"),\n","    \"animals\":   ([\"dog\",\"cat\",\"fish\",\"bird\"],          \"forestgreen\"),\n","    \"cities\":    ([\"paris\",\"london\",\"berlin\",\"rome\"],   \"darkorange\"),\n","    \"countries\": ([\"france\",\"england\",\"germany\",\"italy\"],\"purple\"),\n","}\n","\n","word_to_coord = {w: coords[i] for i, w in enumerate(words)}\n","\n","for group_name, (group_words, color) in groups.items():\n","    xs = [word_to_coord[w][0] for w in group_words]\n","    ys = [word_to_coord[w][1] for w in group_words]\n","    ax.scatter(xs, ys, color=color, s=80, label=group_name, zorder=3)\n","    for w in group_words:\n","        x, y = word_to_coord[w]\n","        ax.annotate(w, (x, y), fontsize=11,\n","                    xytext=(5, 4), textcoords='offset points')\n","\n","ax.axhline(0, color='lightgrey', linewidth=0.8)\n","ax.axvline(0, color='lightgrey', linewidth=0.8)\n","ax.legend(loc='best', fontsize=10)\n","ax.set_title(\"GloVe 50-d word vectors projected to 2D via PCA\", fontsize=13)\n","ax.set_xlabel(f\"PC1 ({pca.explained_variance_ratio_[0]*100:.1f}% variance)\")\n","ax.set_ylabel(f\"PC2 ({pca.explained_variance_ratio_[1]*100:.1f}% variance)\")\n","plt.tight_layout()\n","plt.show()"],"id":"bSjb19rZV0ok"},{"cell_type":"markdown","metadata":{"id":"yzwK_Ad0V0ok"},"source":["### ❓ Question 4\n","\n","> **Look at the 2D plot. Do semantically related words cluster together? Do you notice any geometric patterns — for example, do city–country pairs point in a consistent direction? What does this suggest about how structure is encoded in the embedding space?**"],"id":"yzwK_Ad0V0ok"},{"cell_type":"code","execution_count":null,"metadata":{"id":"zQEAdj47V0ok"},"outputs":[],"source":["# YOUR OBSERVATION: ..."],"id":"zQEAdj47V0ok"},{"cell_type":"markdown","metadata":{"id":"XBT6axwsV0ok"},"source":["---\n","## Part 7 — From Word Vectors to LLMs: The Big Picture\n","\n","Everything we have done so far — one-hot encoding, dense embeddings, similarity, analogy — is the **input layer** of a modern LLM.\n","\n","Here is how the full pipeline looks:\n","\n","```\n","Text input\n","   ↓\n","Tokenisation  (split text into tokens)\n","   ↓\n","Embedding lookup  (token index → dense vector)   ← YOU JUST DID THIS\n","   ↓\n","Transformer layers  (layers of attention + feed-forward)\n","   ↓\n","Output layer  (predict next token probabilities)\n","```\n","\n","Modern LLMs like GPT-4 use **context-sensitive** embeddings: the vector for the word `\"bank\"` differs depending on whether the surrounding context is about finance or rivers. This is handled by the attention mechanism.\n","\n","But the foundation — mapping words to vectors — is exactly what you explored today."],"id":"XBT6axwsV0ok"},{"cell_type":"code","execution_count":null,"metadata":{"id":"WqRmCSXkV0ok"},"outputs":[],"source":["# A tiny demo: the full pipeline in miniature\n","# We represent a short sentence as the SUM of its word vectors (a bag-of-words model)\n","\n","def sentence_vector(sentence, model):\n","    \"\"\"Very simple sentence representation: average of word vectors.\"\"\"\n","    words_in = [w for w in sentence.lower().split() if w in model]\n","    if not words_in:\n","        return None\n","    return np.mean([model[w] for w in words_in], axis=0)\n","\n","sentences = [\n","    \"the cat sat on the mat\",\n","    \"a dog lay on the floor\",\n","    \"the stock market crashed today\",\n","    \"investors lost billions of dollars\",\n","]\n","\n","print(\"Sentence similarities (averaged word vectors):\\n\")\n","vecs = [sentence_vector(s, glove) for s in sentences]\n","\n","for i in range(len(sentences)):\n","    for j in range(i+1, len(sentences)):\n","        sim = cosine_similarity(vecs[i], vecs[j])\n","        print(f\"  [{i+1}] vs [{j+1}]  sim={sim:.3f}\")\n","        print(f\"       '{sentences[i]}'\")\n","        print(f\"       '{sentences[j]}'\")\n","        print()"],"id":"WqRmCSXkV0ok"},{"cell_type":"markdown","metadata":{"id":"BVoVWc0eV0ok"},"source":["### ❓ Question 5\n","\n","> **In the sentence similarity demo, we just averaged the word vectors. Do you think this is a good representation for a sentence? What kind of information would be lost by averaging? What would a more sophisticated model (like a Transformer) do differently?**"],"id":"BVoVWc0eV0ok"},{"cell_type":"code","execution_count":null,"metadata":{"id":"snXbpz_GV0ol"},"outputs":[],"source":["# YOUR ANSWER: ...\n","\n","# Bonus: try your own sentences and see if semantically similar pairs score higher\n","my_s1 = \"I love eating pizza\"\n","my_s2 = \"pasta is my favourite food\"\n","my_s3 = \"the economy is growing\"\n","\n","v1 = sentence_vector(my_s1, glove)\n","v2 = sentence_vector(my_s2, glove)\n","v3 = sentence_vector(my_s3, glove)\n","\n","print(f\"sim(s1, s2) = {cosine_similarity(v1, v2):.3f}  (both about food)\")\n","print(f\"sim(s1, s3) = {cosine_similarity(v1, v3):.3f}  (different topics)\")"],"id":"snXbpz_GV0ol"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.10.0"},"colab":{"provenance":[]}},"nbformat":4,"nbformat_minor":5}