One of the most surprising things about modern AI is that it doesn’t actually work with words—not in the way most people imagine.

When we interact with a chatbot, it feels like the system is reading language directly. We type words. It responds with words. The entire interaction appears linguistic.

But underneath the surface, something very different is happening.

Neural networks work with numerical representations.

So how does a machine turn something as rich and messy as language into numbers that a neural network can actually process?

The answer involves vectors, embeddings, and learned representations.

Why simple numbers aren’t enough

At first, the solution seems obvious.

Assign every word a number:

Cat = 1
Dog = 2
Car = 3

Problem solved.

Except it isn’t.

The moment you represent words this way, you’ve accidentally created relationships that don’t actually exist.

The model could treat “dog” as being closer to “cat” than “car” simply because 2 is closer to 1 than 3.

But those numerical distances are arbitrary.

The numbers contain structure.

The meaning doesn’t.

This reveals an important principle in machine learning:

Converting information into numbers isn’t enough. The numerical representation must preserve useful relationships.

The geometry of meaning: How neural networks represent meaning?

One reason AI can feel mysterious is that meaning itself is difficult to represent.

Humans don’t organize concepts as isolated labels. We understand them through relationships.

A cat is related to a dog.
A king is related to a queen.
A car is more similar to a truck than it is to a banana.

Meaning emerges through connections and context, not simply through labels.

So if a machine is going to work effectively with language, it needs numerical representations that capture useful relationships between concepts.

This leads to the idea of a vector space.

Instead of assigning each word a single arbitrary number, a model can represent it as a vector—a collection of numbers that places it at a particular location in a high-dimensional space.

Concepts with similar patterns of use can end up with representations that are closer together.

What is an embedding?

An embedding is a learned numerical representation of an object, such as a word, token, sentence, image, or other piece of information.

For language, embeddings allow words or tokens to be represented as vectors.

For example, instead of:

cat = 1

a simplified representation might look conceptually like:

cat = [0.21, -0.73, 0.44, ...]

The individual numbers do not have simple human-readable meanings such as “animal” or “small.”

What matters is the pattern of relationships between representations.

If two words are used in similar contexts, their learned representations may capture that similarity.

This gives the model a numerical space in which relationships can be represented and processed.

Learning representations instead of designing them

Earlier AI systems often relied more heavily on humans to manually design useful representations.

Modern neural networks can learn representations from data.

Instead of deciding in advance exactly what each word should mean numerically, the model adjusts its internal parameters during training so that the representations become useful for the task it is learning.

Every word or token can therefore be represented by a collection of numbers.

Not a single number.

A position in a much larger space.

During training, those representations can change as the model learns patterns from its data.

Over time, representations can capture increasingly useful relationships.

The representation reshapes itself around patterns found in the data.

Meaning emerges from relationships

Consider words such as:

• King
• Queen
• Man
• Woman

A model isn’t necessarily given an explicit rule saying exactly how these concepts relate.

Instead, it encounters these words in many different contexts during training.

Patterns of usage provide information about their relationships.

The resulting representations can capture some of those relationships in numerical form.

This is one of the fascinating ideas behind learned representations:

Useful semantic structure can emerge from patterns in data rather than being explicitly programmed as definitions.

The model does not need a dictionary entry inside its parameters that says:

“A king is a male monarch.”

Instead, relationships associated with the concept can be represented across the model’s learned parameters.

The important point is that meaning in a neural network is represented through patterns and relationships, not simply stored as human-readable definitions.

Why representation matters

This idea has a broader consequence.

A neural network can only learn useful patterns if the information it receives is represented in a form that allows those patterns to be captured.

A poor representation can hide important relationships.

A useful representation can make those relationships easier for the network to learn.

This is why representation is such a fundamental idea in machine learning.

The question isn’t simply:

“What algorithm should we use?”

It is also:

“How should the information be represented so the model can learn from it?”

That question becomes particularly important when dealing with language, where relationships between words and their surrounding context are central to meaning.

Beyond words: representations across AI

The same basic idea appears throughout modern AI.

Images begin as pixels.

Sounds can begin as waveforms.

Videos can be represented as sequences of frames.

Raw data enters a neural network as numerical information.

Through successive transformations, the network can build representations that are increasingly useful for the task.

At a simplified level:

Pixels → visual features → objects

Audio → patterns → higher-level representations

Tokens → representations → relationships

The exact mechanisms differ between systems, but the broader pattern is similar.

Raw information is transformed into representations that make useful patterns easier for the network to model.

Representation is not the same as understanding

There is an important distinction here.

When we say that a model represents a concept, that does not mean it understands the concept in exactly the same way a human does.

A vector is not a dictionary definition.

A location in vector space is not a complete theory of meaning.

Instead, these representations capture statistical patterns and relationships that are useful for the model’s tasks.

This distinction matters because it prevents us from turning a useful metaphor into a stronger claim than the technology supports.

The model can develop representations that behave in ways associated with semantic understanding without necessarily possessing human-like understanding.

The bigger shift

The deeper I look at modern AI, the more it feels like a story about representation.

Not intelligence.

Not reasoning.

Representation.

Before a neural network can learn useful patterns, information has to be presented in a form that allows those patterns to be captured.

A better representation can make useful relationships easier to model.

A poor representation can make them harder to detect.

This idea becomes especially powerful when a model needs to understand relationships between many elements of its input.

And that raises the next question:

What kind of neural-network architecture can efficiently model relationships between different parts of a sequence?

That is where the Transformer enters the story.