Word2Vec and Embeddings

In [1]:
%pip install gensim seaborn numpy
Collecting gensim
  Downloading gensim-4.4.0-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl.metadata (8.4 kB)
Requirement already satisfied: seaborn in /usr/local/lib/python3.13/dist-packages (0.13.2)
Requirement already satisfied: numpy in /usr/local/lib/python3.13/dist-packages (2.1.3)
Requirement already satisfied: scipy>=1.7.0 in /usr/local/lib/python3.13/dist-packages (from gensim) (1.16.3)
Requirement already satisfied: smart_open>=1.8.1 in /usr/local/lib/python3.13/dist-packages (from gensim) (8.0.1)
Requirement already satisfied: pandas>=1.2 in /usr/local/lib/python3.13/dist-packages (from seaborn) (2.2.3)
Requirement already satisfied: matplotlib!=3.6.1,>=3.4 in /usr/local/lib/python3.13/dist-packages (from seaborn) (3.10.0)
Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.13/dist-packages (from matplotlib!=3.6.1,>=3.4->seaborn) (1.3.3)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.13/dist-packages (from matplotlib!=3.6.1,>=3.4->seaborn) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.13/dist-packages (from matplotlib!=3.6.1,>=3.4->seaborn) (4.63.0)
Requirement already satisfied: kiwisolver>=1.3.1 in /usr/local/lib/python3.13/dist-packages (from matplotlib!=3.6.1,>=3.4->seaborn) (1.5.0)
Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.13/dist-packages (from matplotlib!=3.6.1,>=3.4->seaborn) (26.3)
Requirement already satisfied: pillow>=8 in /usr/local/lib/python3.13/dist-packages (from matplotlib!=3.6.1,>=3.4->seaborn) (11.3.0)
Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.13/dist-packages (from matplotlib!=3.6.1,>=3.4->seaborn) (3.3.2)
Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.13/dist-packages (from matplotlib!=3.6.1,>=3.4->seaborn) (2.9.0.post0)
Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.13/dist-packages (from pandas>=1.2->seaborn) (2025.2)
Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.13/dist-packages (from pandas>=1.2->seaborn) (2026.3)
Requirement already satisfied: wrapt in /usr/local/lib/python3.13/dist-packages (from smart_open>=1.8.1->gensim) (2.3.0)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.13/dist-packages (from python-dateutil>=2.7->matplotlib!=3.6.1,>=3.4->seaborn) (1.17.0)
Downloading gensim-4.4.0-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl (27.8 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 27.8/27.8 MB 25.7 MB/s eta 0:00:00
Installing collected packages: gensim
Successfully installed gensim-4.4.0

Load pre-trained glove vectors

Subtask:

Load a pre-trained GloVe model to represent words as vectors.

In [2]:
import gensim.downloader as api
import pprint as pp

# List available models to find a suitable GloVe model
pp.pprint(list(api.info()['models'].keys()))
['fasttext-wiki-news-subwords-300',
 'conceptnet-numberbatch-17-06-300',
 'word2vec-ruscorpora-300',
 'word2vec-google-news-300',
 'glove-wiki-gigaword-50',
 'glove-wiki-gigaword-100',
 'glove-wiki-gigaword-200',
 'glove-wiki-gigaword-300',
 'glove-twitter-25',
 'glove-twitter-50',
 'glove-twitter-100',
 'glove-twitter-200',
 '__testing_word2vec-matrix-synopsis']

Reasoning: From the list of models, 'glove-wiki-gigaword-50' seems suitable for demonstration purposes due to its smaller size. The next step is to load this model.

In [3]:
# Load the chosen pre-trained GloVe model
glove_model = api.load('glove-wiki-gigaword-50')

# Optional: Check if the model is loaded correctly
print(f"Model loaded: {glove_model}")
[==================================================] 100.0% 66.0/66.0MB downloaded
Model loaded: KeyedVectors<vector_size=50, 400000 keys>

Reasoning: The task is to load the data from a CSV file into a pandas DataFrame and display the first few rows.

Explore Word Vectors

Subtask:

Show examples of word vectors for a few words.

Reasoning: Word vectors are numerical representations of words. Each word is represented as a point in a multi-dimensional space. Let's look at the vector for the word "king" as an example.

In [4]:
# Get the vector for a word
king_vector = glove_model['king']

# Print the vector
print("Vector for 'king':")
print(king_vector)

# You can also check the dimension of the vector
print(f"\nDimension of the vector: {glove_model.vector_size}")
Vector for 'king':
[ 0.50451   0.68607  -0.59517  -0.022801  0.60046  -0.13498  -0.08813
  0.47377  -0.61798  -0.31012  -0.076666  1.493    -0.034189 -0.98173
  0.68229   0.81722  -0.51874  -0.31503  -0.55809   0.66421   0.1961
 -0.13495  -0.11476  -0.30344   0.41177  -2.223    -1.0756   -1.0783
 -0.34354   0.33505   1.9927   -0.04234  -0.64319   0.71125   0.49159
  0.16754   0.34344  -0.25663  -0.8523    0.1661    0.40102   1.1685
 -1.0137   -0.21585  -0.15155   0.78321  -0.91241  -1.6106   -0.64426
 -0.51042 ]

Dimension of the vector: 50
In [5]:
glove_model.most_similar('king')
Out[5]:
[('prince', 0.8236179351806641),
 ('queen', 0.7839043140411377),
 ('ii', 0.7746230363845825),
 ('emperor', 0.7736247777938843),
 ('son', 0.766719400882721),
 ('uncle', 0.7627150416374207),
 ('kingdom', 0.7542160749435425),
 ('throne', 0.7539913654327393),
 ('brother', 0.7492411136627197),
 ('ruler', 0.7434253692626953)]

Word Analogy Task

Subtask:

Perform the word analogy "king is to man as woman is to _".

Reasoning: One of the fascinating properties of word vectors is their ability to capture relationships between words through vector arithmetic. The classic example is the analogy "king is to man as woman is to _". We can represent this relationship as:

vector("king") - vector("man") + vector("queen") ≈ vector("woman")

Similarly, for the analogy "king is to queen as prince is to _", we can perform the operation:

vector("king") - vector("queen") + vector("prince")

We expect the resulting vector to be closest to the vector for "princess". Let's calculate this and find the words with the highest cosine similarity to the resulting vector.

In [19]:
def word_analogy(model, x, y, z):
  # and negatively correlated with 'x'.
  result = model.most_similar(positive=[y, z], negative=[x], topn=10)
  return result
In [7]:
word_analogy(glove_model, 'king', 'queen', 'man')
Out[7]:
[('woman', 0.8947036266326904),
 ('girl', 0.8489534854888916),
 ('her', 0.7932074666023254),
 ('boy', 0.7824062705039978),
 ('she', 0.770073413848877),
 ('herself', 0.7650615572929382),
 ('mother', 0.7333948612213135),
 ('blind', 0.7331175208091736),
 ('blonde', 0.7081400752067566),
 ('old', 0.7048675417900085)]
In [20]:
word_analogy(glove_model, 'king', 'man', 'queen')
Out[20]:
[('woman', 0.8947036266326904),
 ('girl', 0.8489534854888916),
 ('her', 0.7932074666023254),
 ('boy', 0.7824062705039978),
 ('she', 0.770073413848877),
 ('herself', 0.7650615572929382),
 ('mother', 0.7333948612213135),
 ('blind', 0.7331175208091736),
 ('blonde', 0.7081400752067566),
 ('old', 0.7048675417900085)]
In [21]:
import numpy as np
word_analogy(glove_model, 'cheese', 'bread', 'wine')
Out[21]:
[('coffee', 0.8305814862251282),
 ('beer', 0.7506722211837769),
 ('drink', 0.7202265858650208),
 ('meal', 0.7070305943489075),
 ('food', 0.7067448496818542),
 ('drinks', 0.6861658096313477),
 ('staple', 0.6852551698684692),
 ('beers', 0.6705225110054016),
 ('champagne', 0.6687118411064148),
 ('gourmet', 0.6633869409561157)]
In [22]:
word_analogy(glove_model, 'apple', 'carrot', 'fruit')
Out[22]:
[('asparagus', 0.7416898608207703),
 ('pickled', 0.7246419787406921),
 ('marinated', 0.7203070521354675),
 ('sauteed', 0.7190843224525452),
 ('carrots', 0.7130289077758789),
 ('bunches', 0.7106444239616394),
 ('lettuce', 0.7061347365379333),
 ('arugula', 0.7046541571617126),
 ('figs', 0.7007205486297607),
 ('cucumber', 0.7003002166748047)]
In [11]:
word_analogy(glove_model, 'moon', 'night', 'sun')
Out[11]:
[('morning', 0.7499969005584717),
 ('afternoon', 0.7460827827453613),
 ('home', 0.7298723459243774),
 ('at', 0.7194185256958008),
 ('p.m.', 0.7070990204811096),
 ('went', 0.6951860189437866),
 ('saturday', 0.694523811340332),
 ('sunday', 0.6925952434539795),
 ('weekend', 0.6882964372634888),
 ('evening', 0.6846269965171814)]
In [12]:
word_analogy(glove_model, 'doctor', 'stethoscope', 'teacher')
Out[12]:
[('p.e', 0.6939313411712646),
 ('rudiments', 0.6864296197891235),
 ('undergrad', 0.6743867993354797),
 ('sixth-grade', 0.6616607308387756),
 ('strumming', 0.6555135846138),
 ('homeroom', 0.6510992646217346),
 ('windthorst', 0.6440606117248535),
 ('oberlin', 0.6404677629470825),
 ('sewing', 0.6382118463516235),
 ('fifth-grade', 0.6355699896812439)]
In [13]:
word_analogy(glove_model, 'doctor', 'man', 'woman')
Out[13]:
[('girl', 0.7501261234283447),
 ('old', 0.7375984191894531),
 ('boy', 0.7203571200370789),
 ('black', 0.7126938700675964),
 ('turned', 0.7071391344070435),
 ('behind', 0.7068800330162048),
 ('standing', 0.7012529373168945),
 ('stands', 0.6982771754264832),
 ('looks', 0.6967435479164124),
 ('her', 0.6906110644340515)]
In [14]:
word_analogy(glove_model, 'spain', 'europe', 'china')
Out[14]:
[('asia', 0.7854616045951843),
 ('global', 0.7586598992347717),
 ('japan', 0.7434436678886414),
 ('beijing', 0.7401906847953796),
 ('korea', 0.7363462448120117),
 ('industry', 0.7329097986221313),
 ('chinese', 0.7249193787574768),
 ('domestic', 0.7234880924224854),
 ('expanding', 0.7216194272041321),
 ('taiwan', 0.7131190299987793)]
In [15]:
word_analogy(glove_model, 'alabama', 'football', 'gonzaga')
Out[15]:
[('juventus', 0.7976078987121582),
 ('fiorentina', 0.7713959217071533),
 ('lippi', 0.7336317300796509),
 ('juve', 0.7215985655784607),
 ('mancini', 0.7178478240966797),
 ('trapattoni', 0.7135431170463562),
 ('parma', 0.7118437886238098),
 ('barcelona', 0.7091454863548279),
 ('macala', 0.708977997303009),
 ('torino', 0.7067590951919556)]
In [16]:
word_analogy(glove_model, 'king', 'queen', 'prince')
Out[16]:
[('princess', 0.854616105556488),
 ('duchess', 0.7671989798545837),
 ('victoria', 0.7387416958808899),
 ('consort', 0.7321662306785583),
 ('empress', 0.701124370098114),
 ('countess', 0.6966910362243652),
 ('elizabeth', 0.6949480175971985),
 ('mistress', 0.6901644468307495),
 ('lady', 0.6772074103355408),
 ('sister', 0.6748592257499695)]

Calculate Cosine Similarity

Subtask:

Demonstrate how cosine similarity is used to measure semantic similarity between words.

Reasoning: Cosine similarity measures the cosine of the angle between two vectors. A higher cosine similarity score indicates that the words are more semantically similar. We can use the most_similar method to find words that are similar to a given word based on their vector representations.

In [17]:
# Find words most similar to 'queen'
similar_to_queen = glove_model.most_similar('queen')

print("Words most similar to 'queen':")
for word, similarity in similar_to_queen:
    print(f"{word}: {similarity:.4f}")
Words most similar to 'queen':
princess: 0.8515
lady: 0.8051
elizabeth: 0.7873
king: 0.7839
prince: 0.7822
coronation: 0.7693
consort: 0.7626
royal: 0.7443
crown: 0.7383
victoria: 0.7286

Visualization

Subtask:

Visualize word vectors in a lower-dimensional space to provide a visual representation of semantic relationships.

Reasoning: To visualize the word vectors, I need to select a set of words, get their vectors, reduce their dimensionality using PCA, and then plot them using seaborn. This requires importing the necessary libraries and performing these steps sequentially.

In [18]:
import numpy as np
from sklearn.decomposition import PCA
from sklearn.cluster import KMeans
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
from matplotlib.patches import Polygon
from scipy.spatial import ConvexHull

# 1. Select a list of words
words_to_visualize = [
    'king', 'queen', 'prince', 'princess', 'man', 'woman',
    'apple', 'banana', 'orange', 'grape',
    'cat', 'dog', 'bird', 'fish', 'horse', 'chemistry', 'physics', 'engineering',
    'doctor', 'nurse', 'surgeon', 'plumber', 'electrician', 'mints', 'thumbtack',
    'pencil', 'crayon', 'eraser', 'gum', 'candy', 'gun', 'sword', 'axe'
]

# 2. Extract the word vectors for the selected words
word_vectors = np.array([glove_model[word] for word in words_to_visualize])

# 3. Reduce the dimensionality of the word vectors using PCA to 2 components
pca = PCA(n_components=2)
reduced_vectors = pca.fit_transform(word_vectors)

# 4. Perform K-Means clustering on the reduced vectors
kmeans = KMeans(n_clusters=7, random_state=42, n_init=10)
clusters = kmeans.fit_predict(reduced_vectors)

# 5. Create a pandas DataFrame from the reduced dimensionality vectors and cluster labels
viz_df = pd.DataFrame(reduced_vectors, columns=['pca_dim_1', 'pca_dim_2'])
viz_df['word'] = words_to_visualize
viz_df['cluster'] = clusters

# 6. Use seaborn to create a scatter plot
plt.figure(figsize=(12, 10))
ax = sns.scatterplot(data=viz_df, x='pca_dim_1', y='pca_dim_2', hue='cluster', palette='viridis', s=100) # Added s for point size

# Add shaded backgrounds for each cluster
for cluster_id in sorted(viz_df['cluster'].unique()):
    cluster_points = viz_df[viz_df['cluster'] == cluster_id][['pca_dim_1', 'pca_dim_2']].values
    if len(cluster_points) > 2: # Need at least 3 points to form a polygon
        try:
            hull = ConvexHull(cluster_points)
            # Get a color from the current palette
            color = sns.color_palette('viridis', n_colors=len(viz_df['cluster'].unique()))[cluster_id]
            polygon = Polygon(cluster_points[hull.vertices], closed=True, fill=True, color=color, alpha=0.2, edgecolor=color)
            ax.add_patch(polygon)
        except:
            # Handle cases where ConvexHull fails (e.g., points are collinear)
            pass


# Label the points in the scatter plot
for i, row in viz_df.iterrows():
    plt.annotate(row['word'], (row['pca_dim_1'], row['pca_dim_2']), textcoords="offset points", xytext=(5,5), ha='left')

# 7. Add titles and labels to the plot
plt.title('Word Vectors Visualized with PCA and K-Means Clustering')
plt.xlabel('PCA Dimension 1')
plt.ylabel('PCA Dimension 2')

# 8. Display the plot
plt.legend(title='Cluster')
plt.grid(True)
plt.show()
/tmp/ipykernel_2698/1280919291.py:47: UserWarning: Setting the 'color' property will override the edgecolor or facecolor properties.
  polygon = Polygon(cluster_points[hull.vertices], closed=True, fill=True, color=color, alpha=0.2, edgecolor=color)
/tmp/ipykernel_2698/1280919291.py:47: UserWarning: Setting the 'color' property will override the edgecolor or facecolor properties.
  polygon = Polygon(cluster_points[hull.vertices], closed=True, fill=True, color=color, alpha=0.2, edgecolor=color)
/tmp/ipykernel_2698/1280919291.py:47: UserWarning: Setting the 'color' property will override the edgecolor or facecolor properties.
  polygon = Polygon(cluster_points[hull.vertices], closed=True, fill=True, color=color, alpha=0.2, edgecolor=color)
/tmp/ipykernel_2698/1280919291.py:47: UserWarning: Setting the 'color' property will override the edgecolor or facecolor properties.
  polygon = Polygon(cluster_points[hull.vertices], closed=True, fill=True, color=color, alpha=0.2, edgecolor=color)
/tmp/ipykernel_2698/1280919291.py:47: UserWarning: Setting the 'color' property will override the edgecolor or facecolor properties.
  polygon = Polygon(cluster_points[hull.vertices], closed=True, fill=True, color=color, alpha=0.2, edgecolor=color)
/tmp/ipykernel_2698/1280919291.py:47: UserWarning: Setting the 'color' property will override the edgecolor or facecolor properties.
  polygon = Polygon(cluster_points[hull.vertices], closed=True, fill=True, color=color, alpha=0.2, edgecolor=color)
/tmp/ipykernel_2698/1280919291.py:47: UserWarning: Setting the 'color' property will override the edgecolor or facecolor properties.
  polygon = Polygon(cluster_points[hull.vertices], closed=True, fill=True, color=color, alpha=0.2, edgecolor=color)
No description has been provided for this image
In [18]:
 

Embedding Phrases

In [18]:
 

What are Word Vectors?

At the core of Word2Vec (and GloVe) are word vectors, also known as word embeddings. Think of these as numerical representations of words. Instead of just having a word as a string of letters, we represent each word as a list of numbers.

Each number in the vector represents some latent feature or characteristic of the word's meaning or context. Words that are used in similar contexts or have similar meanings tend to have similar vectors.

In our case, we are using a pre-trained GloVe model, specifically "glove-wiki-gigaword-50". This means each word is represented by a vector with 50 dimensions. While we can't easily visualize 50 dimensions, the idea is that these 50 numbers capture the semantic essence of the word.

Let's look at the vector for the word "king" again:

Cosine Similarity

So, we have these word vectors. How do we use them to figure out if two words are similar in meaning? This is where cosine similarity comes in.

Cosine similarity is a metric that measures the cosine of the angle between two vectors.

  • If the angle between two vectors is 0 degrees (they point in the exact same direction), the cosine similarity is 1. This means the words are highly similar.
  • If the angle is 90 degrees (they are perpendicular), the cosine similarity is 0. This means there is no similarity.
  • If the angle is 180 degrees (they point in opposite directions), the cosine similarity is -1. This indicates strong dissimilarity.

In simpler terms, cosine similarity tells us how similar the direction of the two word vectors is. Words with similar meanings tend to have vectors that point in similar directions in the vector space, resulting in a higher cosine similarity score.

We can use the most_similar method of the GloVe model to find words that have the highest cosine similarity to a given word. This method essentially calculates the cosine similarity between the given word's vector and all other word vectors in the model and returns the words with the highest scores.

Let's look at the words most similar to "queen" again:

Word Analogies and Vector Arithmetic

One of the most surprising and powerful demonstrations of word vectors is their ability to capture relationships between words using simple vector arithmetic. The famous example is the analogy "king is to queen as prince is to _".

In the vector space, the relationship between "king" and "queen" (which is related to gender) is captured by the vector difference:

vector("king") - vector("queen")

This resulting vector represents the "maleness" or "femaleness" relationship.

To solve the analogy "king is to queen as prince is to _", we can perform a similar operation, but starting with "prince" and applying the "king" to "queen" transformation. We subtract the vector of the word we are moving from in the first pair and add the vector of the word we are moving to:

vector("prince") - vector("king") + vector("queen")

The resulting vector should be very close to the vector for the word that completes the analogy, which is "princess".

The most_similar method in gensim allows us to perform this kind of vector arithmetic directly by providing positive and negative words. The method calculates the vector sum of the positive words and subtracts the vector sum of the negative words, then finds the words with the highest cosine similarity to the resulting vector.

We performed the analogy "king is to queen as prince is to _" using positive=['prince', 'queen'] and negative=['king']. As expected, "princess" was the top result, demonstrating that the model learned this relationship.

Visualizing Word Vectors with PCA

As we saw, word vectors can have many dimensions (in our case, 50). It's impossible for us to directly visualize vectors in such high-dimensional space. To get a visual intuition of how these vectors are organized and how they capture semantic similarity, we can use dimensionality reduction techniques.

Principal Component Analysis (PCA) is a common technique used for this purpose. PCA finds the directions (principal components) in the high-dimensional space that capture the most variance in the data. By projecting the vectors onto the first few principal components (in our case, the first two), we can represent them in a lower-dimensional space (like 2D) while trying to preserve as much of the original relationships as possible.

The resulting 2D plot gives us a visual representation of the word vectors. Words that are semantically similar are expected to be clustered together in this reduced-dimensional space.

In the scatter plot we generated:

  • Each point represents a word.
  • The position of the point is determined by the word's vector projected onto the first two principal components.
  • Words that appear close together on the plot are likely to be semantically similar, even though we reduced the dimensionality from 50 to 2.

Looking at the plot, you should observe that words like "king", "queen", "prince", and "princess" are clustered together, as are the fruit names ("apple", "banana", "orange", "grape") and the animal names ("cat", "dog", "bird", "fish"). This visually reinforces the idea that word vectors capture semantic meaning and relationships.

Summary:

Data Analysis Key Findings

  • Pre-trained GloVe word vectors successfully represent semantic relationships, as demonstrated by the clustering of related words (royalty, fruits, animals) in the 2D PCA visualization.
  • PCA effectively reduces the dimensionality of word vectors (from 50 to 2) while largely preserving the semantic structure, making visualization possible.
  • Word vectors allow for the representation of analogies through vector arithmetic, as shown by the "man - king + queen" calculation yielding "woman" as the most similar word.
  • Cosine similarity is a useful metric for quantifying the semantic similarity between word vectors, with higher scores indicating closer meanings.