On Multimodal Models

2026-08-25T00:00:00Z

On Multimodal Models

What Should a Universal Multimodal Representation Look Like?

Multimodal AI is converging on a deceptively simple idea: text, images, audio, video, and actions should somehow work inside the same model.

But “the same model” and “the same representation” are not the same thing.

Today's systems can process several modalities natively, but they still need ways to make very different kinds of information compatible. Projectors, modality-specific encoders, language intermediaries, discrete tokenizers, contrastive losses, temporal pooling, and specialized action heads all solve parts of the problem.

That leaves a more interesting question:

What information should actually be shared, at what precision, and in what form?

I don't think the answer is that everything should become tokens. I also don't think everything should be compressed into one giant float vector.

A more plausible design is a mixture of shared world state, discrete semantic events, continuous state, and modality-specific residual information.

1. The Multimodal Frontier

There are several different things people mean when they talk about “multimodal AI.”

Unified Embedding Space

The simplest goal is to put different modalities into a common space:

text
image
audio
video
document
   ↓
one vector space

One prominent example is Gemini Embedding 2, which maps text, images, video, audio, and documents into a shared embedding space for cross-modal retrieval and similarity.

That's useful, but it is worth being precise about what it demonstrates.

A shared embedding space is not necessarily a shared reasoning space.

If a video of someone playing guitar and the sentence “someone is playing guitar” end up close together, the model has learned semantic alignment. It does not follow that the model reasons about the video using the same internal representation it uses for the sentence.

2. Native Omni Models

The next step is to process several modalities inside one generative system.

Models such as Qwen3-Omni are examples of this direction. Text, images, audio, and video can enter the same system, and the model can also generate speech.

The interesting part is that “one model” does not necessarily mean “one encoder.” Modern omni systems still use modality-specific components while sharing a large amount of computation.

Conceptually:

                 ┌── image
                 │
                 ├── video
WORLD ───────────┼── audio
                 │
                 └── text
                       ↓
              multimodal model
                       ↓
             reasoning / generation

That's a stronger form of integration than simply concatenating inputs. Still, the modalities do not have to be represented identically.

And that may actually be a good thing.

3. Latent Multimodal Reasoning

A useful research question is whether a model should reason directly over sensory representations instead of translating everything into language first.

A common pipeline looks like:

audio ──→ transcript ──┐
video ──→ description ─┼──→ LLM reasoning
image ──→ caption ────┘

This works remarkably well because language is a powerful abstraction.

It is also a bottleneck.

Compare:

“The glass starts falling.”

with the underlying physical trajectory:

[x(t), y(t), z(t), v(t), a(t)]

The sentence captures the event. It doesn't preserve the metric dynamics.

Recent work such as LatentOmni explores reasoning with audio-visual latent representations rather than forcing sensory information through textual chain-of-thought.

The distinction matters:

Language is an excellent interface to thought. It may not be the best internal representation for every kind of multimodal reasoning.

4. VLA Makes the Problem Harder

Vision-language-action models make the limits of language-centric multimodality much easier to see.

A command such as:

“Pick up the cup.”

is easy to represent linguistically.

A robot needs much more:

object pose
camera geometry
joint positions
velocity
contact state
grasp configuration
forces
trajectory
uncertainty
physical dynamics

Its relevant state looks more like:

[q(t), q̇(t), τ(t), x_object(t)]

than a sequence of words.

This is why systems such as π0/π0.5 and Gemini Robotics retain specialized machinery for action generation.

Robotics changes the question. It is no longer enough to make two modalities semantically similar. The representation has to connect semantics to physical state and dynamics.

5. Floats or Bits?

One of the basic representation choices is whether everything should become:

z ∈ R^d

or:

[t₁, t₂, ..., tₙ]

The first gives us continuous vectors. The second gives us discrete tokens.

Both are useful. Neither seems sufficient by itself.

5.1 Continuous Representations

Continuous latent vectors are good at representing semantic geometry.

A system can arrange things so that:

dog image
dog video
dog bark
"dog"

occupy nearby regions of a latent space.

That's useful for:

retrieval
semantic similarity
alignment
interpolation
differentiable computation
multimodal fusion

There is a catch, though.

Embeddings are supposed to throw information away.

For retrieval, different observations of the same thing should often become similar:

dog #1
dog #2
dog #3

But a generative or physical model may need to know exactly how those dogs differ.

The representation that is ideal for retrieval is therefore not necessarily the representation needed for reconstruction, simulation, or control.

5.2 Discrete Tokens

The other obvious approach is to tokenize everything:

TEXT    → text tokens
IMAGE   → visual tokens
AUDIO   → audio tokens
VIDEO   → video tokens
ACTION  → action tokens

Now a Transformer has one convenient interface.

That is attractive. The model can process something like:

[text][image][audio][video][action]

using largely the same machinery.

But there is a conceptual problem:

A token is a serialization format. It isn't necessarily a natural representation of the thing being serialized.

Audio makes this obvious.

A semantic audio representation might look like:

[SPEECH]
[HELLO]
[CAR_PASSING]
[LAUGHTER]

But that throws away pitch, timbre, phase, speaker identity, room acoustics, exact timing, spatial structure, and subtle prosody.

You can raise the token rate to preserve more of the signal. At some point, though, the universal token representation starts looking like an expensive codec.

6. A More Plausible Answer: Hybrid Representation

A more practical design might combine:

continuous latent
+
discrete events
+
modality-specific residual

Think of the representation as having different information rates.

Semantic layer

Discrete concepts or events:

DOG
BALL
SPEECH
GRASP
DOOR_OPEN
PERSON_WALKING

State layer

Continuous quantities:

position
velocity
orientation
depth
pose
speaker state
robot joint state

Residual layer

High-bandwidth details:

pixel texture
waveform detail
fine motion
timbre
lighting
sensor noise

The semantic layer can be heavily compressed.

Physical state often needs numerical precision.

And there is information that simply should not be discarded just because it is inconvenient to align across modalities.

7. A Possible Universal Representation

Instead of forcing every modality into one vector, consider:

z_m = [
    z_world,
    z_task,
    z_modal,m,
    z_residual,m
]

where:

`z_world`

Information that can be shared across modalities:

entities
objects
events
relationships
concepts
spatial structure
causal structure

`z_task`

Information useful across tasks such as:

reasoning
planning
retrieval
instruction following

`z_modal`

Information that is specific to a modality:

acoustic phase
optical texture
phonetic detail
optical flow
tactile patterns
motor dynamics

`z_residual`

High-bandwidth information needed for faithful reconstruction.

This gives us a slightly different definition of “unified representation.”

The modalities do not need to become identical. They need to become different observations of a shared latent world.

8. The Shortcuts Current Models Use

Current multimodal models are impressive, but many depend on architectural shortcuts to make different modalities work together.

I don't mean “shortcut” as an insult here. These are mostly good engineering decisions. They are worth examining because they also show where the representation problem is still open.

8.1 The Projector

A classic architecture looks like:

image
  ↓
vision encoder
  ↓
visual features
  ↓
MLP / projector
  ↓
LLM embedding space
  ↓
Transformer

The projector learns to translate one model's output into something another model can consume.

That gives us interface compatibility.

It doesn't necessarily give us representation unification.

The vision encoder and language model may still have very different internal structures.

8.2 The “Everything Is Language” Shortcut

This is probably the most powerful shortcut of all.

image ──→ description ──┐
audio ──→ transcript ───┼──→ LLM
video ──→ caption ──────┘

The problem becomes easier because language acts as a common intermediary:

image → language → reasoning
audio → language → reasoning

It works because language packs an enormous amount of semantic information into a compact form.

But the translation has a cost.

Exact temporal, spatial, acoustic, and physical information can disappear simply because it is awkward to express in language.

8.3 Contrastive Alignment

Another common strategy is to pull representations from different modalities closer together.

Conceptually:

             "dog"
            /  |  \
           /   |   \
       image audio video

The model learns something like:

z_image ≈ z_text ≈ z_audio ≈ z_video

That's very effective for semantic retrieval.

But the objective is usually closer to:

These observations should be semantically similar.

It is not necessarily:

These representations should have the same internal computational structure.

That difference becomes important once the model has to do more than retrieve or classify.

8.4 The Discretization Shortcut

Another clean solution is to turn every modality into tokens:

image → visual tokens
audio → audio tokens
video → video tokens
action → action tokens

Then one Transformer can process everything:

[text]
[image]
[audio]
[video]
[action]
   ↓
Transformer

The downside is that tokenization imposes a vocabulary and a granularity on the underlying signal.

An image is not naturally a sequence of vocabulary IDs.

Neither is a waveform.

Tokens are useful because they provide a convenient interface for autoregressive models. That doesn't mean reality itself is discrete.

9. The Specialist-Encoder Approach

The opposite strategy is to keep separate experts:

          ┌── vision encoder
          │
          ├── audio encoder
          │
WORLD ────┼── video encoder
          │
          ├── language model
          │
          └── action expert

Adapters and cross-attention connect them.

In many cases, this is probably the right engineering decision.

Different modalities really do have different statistical structures. A waveform does not have the same local structure as a sentence. A robot trajectory has a different structure again.

So perhaps the goal should not be to eliminate modality-specific experts.

The harder problem is figuring out how those specialized representations communicate through a shared latent world model.

10. Temporal Pooling Is Another Compression Trick

Video makes the problem even more obvious.

At 30 FPS, one minute contains:

30 × 60 = 1800 frames

At useful resolution, processing all of them is expensive.

So systems compress:

1800 frames
     ↓
vision encoder
     ↓
temporal pooling
     ↓
hundreds of tokens
     ↓
LLM

The model is effectively being asked to keep the useful information and throw away the rest.

That is fine for many semantic questions.

It becomes less comfortable when the task depends on:

exact timing
subtle motion
physical dynamics
causality
long-horizon interaction
precise action

This is one reason long-video modeling and multimodal compression are still active areas of research.

11. Language as a Universal Hub

There is another version of the same idea.

Suppose direct image-to-audio alignment is difficult.

Instead:

image → language
audio → language

and then:

image
  ↕
language
  ↕
audio

Language becomes the hub.

That is a very effective architecture for assistants. It also creates a strong bias: modalities are treated as equivalent to the extent that language can describe their equivalence.

That is probably not enough for a complete model of the physical world.

12. What VLA Reveals

Robotics makes this distinction hard to ignore.

A language representation might contain:

PICK_UP(cup)

The robot needs something more like:

cup_pose
arm_pose
joint_velocity
gripper_state
contact_state
trajectory
force
uncertainty

The first is semantic.

The second is physical state.

You can encode the second as tokens if you really want to, but that doesn't make tokens the natural representation of the underlying problem.

This is why modern VLA systems often retain specialized action policies or action experts.

The architecture is effectively telling us something:

Action is not just another language modality. It is tied to dynamics.

13. What Does “Unified” Actually Mean?

It helps to separate a few different levels.

Level 0 — Separate Models

vision model
audio model
language model
robot model

Level 1 — Shared Semantic Embedding

image ─┐
audio ─┼──→ R^d
text ──┤
video ─┘

Excellent for retrieval and alignment.

Level 2 — Shared Latent Reasoning

image
audio
video
text
  ↓
shared latent state
  ↓
reasoning

The model can reason without translating every observation into language.

Level 3 — Shared World Model

                 WORLD
                   ↓
            latent dynamics
                   ↓
       ┌───────────┼───────────┐
       ↓           ↓           ↓
     speech       video       action

This is the difficult one.

A useful world representation would need to capture things like:

state
dynamics
causality
uncertainty
geometry
temporal structure
affordances

The field is much closer to Level 1 than Level 3.

14. Why One Euclidean Space May Be the Wrong Goal

Suppose every modality has to produce:

z_text,
z_image,
z_audio,
z_video,
z_action ∈ R^d

What should coordinate 173 mean?

Is it redness?

Phoneme intensity?

Semantic importance?

Robot torque?

Spatial depth?

Trying to make every dimension universal seems like the wrong constraint.

A more natural formulation is:

z_m = [
    z_shared,
    z_private,m
]

The shared space captures what the modalities have in common.

The private space keeps what they fundamentally don't.

This is closer to how sensors work in the real world. A camera, microphone, tactile sensor, and joint encoder all observe different projections of the same underlying environment.

They should not produce identical observations.

They should help us infer a common state.

15. The Representation Worth Betting On

A future multimodal foundation model could look something like this:

                    WORLD
                      │
       ┌──────────────┼──────────────┐
       ↓              ↓              ↓
     vision         audio          sensors
       │              │              │
       └──────────────┼──────────────┘
                      ↓
              modality encoders
                      ↓
          ┌─────────────────────┐
          │    SHARED WORLD     │
          │        LATENT       │
          │                     │
          │ semantic state      │
          │ geometry            │
          │ dynamics            │
          │ uncertainty         │
          └──────────┬──────────┘
                     │
             ┌───────┴───────┐
             ↓               ↓
       discrete events   continuous state
             │               │
             └───────┬───────┘
                     ↓
                WORLD MODEL
                     │
        ┌────────────┼────────────┐
        ↓            ↓            ↓
      text         speech       action

The model would not need to preserve the fact that every piece of information started life as a pixel, waveform, word, or motor command.

It would reason over a representation of the world.

That does not mean throwing away the original modalities. It means giving them somewhere meaningful to meet.

16. The Fundamental Unit May Be an Event

There is another possibility that I find more interesting than either “token” or “vector.”

Maybe the useful primitive is an event or state transition.

For example:

ENTITY
    cup_17

STATE
    position = (x, y, z)
    orientation = (...)
    velocity = (...)

RELATION
    cup_17 in_front_of person_2

EVENT
    person_2 grasps cup_17

UNCERTAINTY
    0.93

That structure can be:

seen
heard
described
touched
manipulated
predicted
remembered

The modalities remain different. What they share is a world that they can observe, change, and reason about.

That feels like a stronger definition of multimodal unification than “everything lives in the same vector space.”

17. Bytes Are Making a Comeback

There are two tempting extremes:

Everything becomes text tokens

Too lossy for some tasks.

Everything becomes one giant float vector

Too invariant for some tasks and not expressive enough for high-fidelity generation or physical action.

A more plausible combination is:

shared latent world state
+
discrete semantic events
+
continuous state
+
modality-specific residuals

The pieces have different jobs:

Representation	Role
Continuous latent	World state, geometry, dynamics, semantic geometry
Discrete tokens/events	Communication, symbolic structure, compression, autoregressive generation
Modality-specific residual	High-bandwidth sensory detail
Specialized heads/experts	Interfaces to speech, images, video, robots, and other modalities

The important distinction is between serialization and representation.

A byte stream is a way to communicate information.

A token sequence is a convenient way to feed information into an autoregressive model.

Neither necessarily describes the natural structure of the information itself.

For years, the main interface between raw data and language models has been the tokenizer.

Text gets converted into BPE, SentencePiece, WordPiece, or another learned vocabulary:

"multimodal"
      ↓
[token_...][token_...]
      ↓
Transformer

This is efficient. Rather than processing every character or byte independently, the model gets units that capture recurring patterns.

But tokenization is also learned compression. The tokenizer decides what the model receives, how long the sequence becomes, where boundaries fall, and which patterns become atomic.

That leads to a slightly provocative question:

What if the model operated directly on bytes instead?

EvaByte: Modeling the Raw Byte Stream

EvaByte is one of the clearer recent attempts to make that practical.

Instead of beginning with a learned subword vocabulary, EvaByte operates directly on bytes. Its approach combines:

EVA attention, intended to make byte-level processing computationally practical
multibyte prediction, allowing the model to predict multiple bytes instead of taking one autoregressive step for every byte

Conceptually:

raw bytes
    ↓
byte embeddings
    ↓
EVA attention
    ↓
latent computation
    ↓
multibyte prediction
    ↓
raw bytes

EvaByte is a 6.5B-parameter byte-level model trained on roughly 1.5T bytes of text, mathematics, and code.

The interesting part is not just that it avoids a tokenizer.

It treats the byte as a first-class computational unit.

18. Where the Frontier Is Heading

The broad trajectory looks something like:

separate modality models
          ↓
shared embeddings
          ↓
LLM + modality adapters
          ↓
native omni models
          ↓
latent multimodal reasoning
          ↓
shared world representations
          ↓
world model + action

The first half is already fairly mature.

The last two stages are where the harder representation problem begins.

The central challenge is no longer:

How can an image encoder communicate with an LLM?

That problem is largely solved.

The harder question is:

How can one latent state account for what is seen, heard, said, remembered, predicted, and physically done without reducing everything to language or throwing away modality-specific information?

I don't think the answer is simply floats vs. bits.

A more plausible direction is floats + bits + structure, organized around a shared model of the world rather than one universal coordinate system.

Blog Title

On Multimodal Models

On Multimodal Models

What Should a Universal Multimodal Representation Look Like?

1. The Multimodal Frontier

Unified Embedding Space

2. Native Omni Models

3. Latent Multimodal Reasoning

4. VLA Makes the Problem Harder

5. Floats or Bits?

5.1 Continuous Representations

Embeddings are supposed to throw information away.

5.2 Discrete Tokens

6. A More Plausible Answer: Hybrid Representation

Semantic layer

State layer

Residual layer

7. A Possible Universal Representation

z_world

z_task

z_modal

z_residual

8. The Shortcuts Current Models Use

8.1 The Projector

8.2 The “Everything Is Language” Shortcut

8.3 Contrastive Alignment

8.4 The Discretization Shortcut

9. The Specialist-Encoder Approach

10. Temporal Pooling Is Another Compression Trick

11. Language as a Universal Hub

12. What VLA Reveals

13. What Does “Unified” Actually Mean?

Level 0 — Separate Models

Level 1 — Shared Semantic Embedding

Level 2 — Shared Latent Reasoning

Level 3 — Shared World Model

14. Why One Euclidean Space May Be the Wrong Goal

15. The Representation Worth Betting On

16. The Fundamental Unit May Be an Event

17. Bytes Are Making a Comeback

Everything becomes text tokens

Everything becomes one giant float vector

EvaByte: Modeling the Raw Byte Stream

18. Where the Frontier Is Heading

First post

`z_world`

`z_task`

`z_modal`

`z_residual`