Learning World Models by Building One From Scratch

Building video world models piece by piece through an open, hands-on community learning series.

A note on authorship: The views and opinions expressed in this post belong to the authors. Cohere Labs supports open sharing, discussion, and dialogue, and is proud to provide a platform for voices and perspectives from our global community. Learn more about our Open Science initiatives.

Intro:

Nahid Alam (https://www.linkedin.com/in/nahidalam/) is a researcher working on multimodal agents, with a focus on how reinforcement learning and world models can help agents understand, predict, and interact with their environments. Previously, she led Maya (github.com/nahidalam/maya), an open multilingual language model developed as a community research project within Cohere Labs.

What I am building

I am building World Models from Scratch, an open, hands-on project for understanding how video world models actually work by building one piece by piece. The goal is to go from an input video to a model that can predict possible future frames, while making every major step visible.

I want the project to be something people can run, modify, break, and learn from.

What is a World Model?

A world model learns how an environment changes over time and predicts what might happen next. It can help AI agents simulate possible futures and plan their actions.

Why I find it exciting

The idea of simulating different possible futures fascinates me, almost like the TV series Person of Interest. I often wonder how a different decision could have changed an outcome, or what life might have looked like if I had taken another path.

Technically, world models offer a way to simulate alternative futures and explore how different actions influence outcomes. This opens up exciting possibilities for counterfactual reasoning, planning, and decision-making in AI agents.

What Sparked it

World models are moving incredibly quickly. We are seeing models that can generate realistic video, simulate environments, support planning and decision-making. These systems feel like black boxes. It is easy to run them without really understanding what is happening inside.

I wanted to be in a different direction.

I wanted to see what happens at every stage. How does a video become something a Transformer can process? What does the model learn during training? How does it predict future frames? How can we make these models small enough for more people to experiment with them?

What I have actually done

We are building the book in the open, publishing as we go https://github.com/nahidalam/world_model_from_scratch

So far, we have run three sessions with the community. These sessions have been one of the most valuable parts of the project. People ask difficult questions, challenge assumptions, give active feedback, and build alongside the material.

The project is becoming a shared learning process rather than something I am building alone.

Why community matters

I hope this project becomes a useful learning resource for the community.

I want others to explore the code, train small models, test different ideas, and share what they discover. World models are developing quickly, and there are many open questions to explore.

Learning together can make these ideas easier to understand, more accessible to people who want to start experimenting with them and hopefully spark a new idea among them!

Where it is going

I hope the project becomes a go to resource for the community. As we learn more, I want to turn those lessons into code and explanations that make world models easier for others to build and understand.

Call to Action:

If you are curious about world models, come build with us.

Try the code and join one of our upcoming sessions https://luma.com/CLEmbodiedAI