Inside the agent: how language models build worlds

You type: "A cottage by a stream, with a mossy roof and a garden." A minute later, blocks start to appear in your browser. It looks as if the agent saw the cottage in its mind and then built it.

That is not quite what happens – and the difference explains almost every design decision in genMC.

What a language model actually sees#

A language model reads and writes tokens: small pieces of text. It has no eyes on your build and no 3D grid inside. When it works in genMC, everything it knows about the build arrives as text – the size of the plot, the list of named parts, the coordinates of your marks – plus, if the model can read images, the pictures genMC renders for it.

Everything it knows about cottages, mossy roofs, stairs and gardens comes from training: from billions of pages of text and code, and for many models also images. Nobody taught it what a gable roof is. It picked it up.

The world model inside#

Does a model only shuffle words, or does it build some inner picture of how the world is arranged? Research suggests it builds more than you might expect.

• In the Othello-GPT study, a small model was trained on nothing but lists of moves from the board game Othello. It never saw a board. Yet researchers could read the current board state out of its internal activations – and when they changed that internal state, the model's next moves changed to match. It had built a board in its head, because that was the best way to predict the next move.
• Gurnee and Tegmark found that large language models carry linear representations of space and time: from a model's internal activity for a place name, you can read off roughly where on Earth the place is, and for an event roughly when it happened.

This is what people mean by an internal world model: a compressed, implicit sense of how things relate, learned because it helps to predict text. For building, it means a model "knows" that a roof sits on walls, that doors face paths, that stairs climb and that trees thin out toward a mountain's snow line. It can imagine a plan.

But this inner model is fuzzy:

• It is approximate. Off-by-one errors, left and right mixed up, a wall one block short – these are typical, because nothing inside the model is an exact grid.
• It is forgetful. The model's working memory is its context window. Fifty steps into a build, the exact contents of row 143 are long gone unless they are written down somewhere.
• It is unverified. The model cannot tell from the inside whether its plan really works – whether a room hangs over a cliff or a corridor is too low to walk through.

Two meanings of "world model"#

The term has a second meaning that is everywhere at the moment. Systems such as DeepMind's Genie 3 or Decart's Oasis are video world models: they learn to predict the next frame of a game from the previous frames and your input. You can walk through worlds that never existed, in real time.

In these systems the world exists only as pixels. There is no list of blocks to edit, nothing to export, and consistency is limited: Genie 3 keeps its world stable for a few minutes, and early versions of Oasis were known to forget what lay behind you once you turned around.

genMC takes the opposite route. The world is explicit: an exact grid of blocks that is saved, versioned, undoable and exportable. The language model's internal world model does what it is good at – imagining, planning, composing – and the explicit model in genMC holds the truth. Neither would be enough alone.

The loop: imagine, write, run, look, fix#

Here is what a good agent does in genMC, step by step.

1. Read. It asks the studio for the build's state and context: the plot size, the existing named parts, your requests and your marks – marked blocks arrive as exact boxes, so your pointing becomes coordinates.
2. Imagine and tell you. It forms a plan – silhouette first, then the main masses, then details – and posts it in the agent column, so you can object before it starts.
3. Write a program. It compresses the plan into a construction program: a loop for each wall, an ellipsoid for each tree crown, noise for a hillside, helpers for stairs and roofs. A few hundred lines of code can place a million blocks. The same blocks as coordinates would take millions of tokens.
4. Run it in the sandbox. genMC runs the program in isolation, with a time limit and no network access. The result is deterministic – same program, same seed, same build – and every placed block must sit inside a named part.
5. Look. The agent renders the result from cameras it chooses, cuts through floors and walls, and looks at the pictures. This is the moment where the fuzzy inner model meets reality: the tower I imagined is thinner than it looks; the path runs into the pond.
6. Check. Some questions have exact answers that are cheap to compute. Which parts hang over lower ground? Where would a tour camera clip through a wall? Can a player walk between the rooms with two blocks of headroom? genMC answers these from the actual blocks, not from the model's memory.
7. Fix. It writes the next program. Every step lands on the timeline, so nothing is lost and you can go back at any time.

Giving the model the right handles#

Most of genMC's work is translation: between how people point and talk, how models think, and how blocks are stored.

• Coordinates spelled out. X points east, Y up, Z south, both corners of a box are included. Ambiguity is where fuzzy models fail, so we remove it.
• Named parts as shared words. Cottage/Moss roof or Castle/East tower is a handle that you, the agent and the studio all understand. The agent can revisit a part fifty steps later without re-deriving the whole scene from its memory.
• Marks become boxes. When you paint over blocks with the marking tool, the agent receives an exact, compact set of boxes, not a screenshot it would have to interpret.
• Colour, measured. "Warm stone fading to pale at the top" is a feeling. genMC measures the colour of every block as it looks in the game and finds the smoothest chain of real blocks between two of them. The model brings the taste; the studio brings the numbers.
• Skins as a map. In the skin studio, every one of the 72 faces has a name, so the agent never has to solve the skin's texture layout as a puzzle.
• Revisions as a reality check. Every change names the revision it was based on. If you edited something in the meantime, the change is rejected and the agent has to look again. Its picture of the build can never be silently out of date.

Why this matters#

Models get better at spatial reasoning every few months. genMC does not bet on a single model. It gives whichever agent you use an exact world, a pair of eyes and a ruler – and lets the model's imagination do the rest. As models improve, your builds improve, in the same studio.

And the result is not a dream that fades when you look away. It is a build made of real blocks that you can walk through, change, and take with you into Minecraft.

Curious how this compares with other AI tools? Read what genMC does differently. The full working rules for agents are in the guide for agents.

Sources#

• Kenneth Li et al.: Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task, ICLR 2023.
• Wes Gurnee, Max Tegmark: Language Models Represent Space and Time, 2023.
• Google DeepMind: Genie 3: A new frontier for world models, 2025.