LeFlow: Generative Latent Flow Planning for World Models

Hsiang-Wei Huang Jianxu Shangguan Junbin Lu Jenq-Neng Hwang
University of Washington
Plan by generating where to go, not by searching over what to do. LeFlow sits on a frozen latent world model (LeWM). It learns a reusable latent trajectory prior offline, then at test time samples latent paths between the current and goal embeddings, decodes them into actions, and lets the world model pick the best one.
Faster than LeWM + CEM 4.5–14.4× end-to-end evaluation time
Benchmarks with best success 4 / 4 TwoRoom, PushT, Reacher, OGBench-Cube
Gain over LeWM + CEM +5.9 to +26.7 success-rate points
World model changes None LeWM stays frozen

Abstract

Latent world models are inherently strong encoders that transform image pixel to latent embedding, yet existing world models still rely on online trajectory optimization for action planning: for every state–goal pair, an iterative optimizer is run from scratch to search for optimal action sequences, treating the world model as a black-box simulator. This approach pays the full iterative optimization cost anew at every replanning step and reuses no planning experience across queries. In this work, we ask whether planning itself can be amortized once a latent world model has been learned. We present LeFlow, which learns a reusable latent trajectory prior operating directly in the latent dynamics space from the world model. LeFlow recasts planning as conditional latent trajectory generation: a rectified-flow model imagines a future latent path between the current and goal embeddings, an inverse dynamics decoder turns latent transitions into action chunks, and the frozen world model verifies each candidate by autoregressive rollout. Across four major goal-conditioned pixel-control benchmarks, LeFlow replaces iterative action-space optimization with amortized latent planning and fixed-budget rollout selection, achieving consistent success-rate gains with an order-of-magnitude reduction in planning time. Our results argue that latent world models should support not only prediction but reusable planning priors.

Method

Plan in latent state space, decode actions afterwards, verify with the world model.

World models such as LeWM plan with the Cross-Entropy Method: for every state–goal pair they search over raw action sequences from scratch, querying the world model thousands of times per replanning step and reusing nothing across queries. Action sequences are low-level and highly multimodal, since many of them realize nearly the same state trajectory. Latent state trajectories are smoother and already organized by the world model's encoder, so LeFlow learns its planning prior in that space instead.

1

Latent flow planner

A conditional rectified-flow model generates the interior of a latent path whose endpoints are clamped to $z_{\text{start}}$ and $z_{\text{goal}}$. Its near-straight transport needs only 16 Euler steps, and $N{=}64$ candidates are sampled in parallel.

2

Inverse dynamics decoder

A lightweight MLP maps each latent transition, together with its displacement, to an executable action chunk: $a_t = g_\omega([z_t,\, z_{t+1},\, z_{t+1}-z_t])$. The planner decides where to go; the decoder decides how.

3

Rollout reranking

Each decoded action sequence is rolled out through the frozen LeWM predictor. The candidate whose actual rollout lands closest to the goal is executed. This projects generated paths back onto the latent manifold the world model can actually control.

Rollout reranking projects generative proposals onto the controllable latent manifold. Generated interiors may pass through latent states no admissible action reaches. Each candidate is decoded into actions and rolled out through the frozen LeWM predictor, then scored by $\lVert \hat z_H^{(i)} - z_{\text{goal}} \rVert_2^2$; the lowest score is executed.

Training objective

LeWM stays frozen. Only the flow planner $v_\psi$ and the decoder $g_\omega$ are trained, jointly, on offline trajectories:

$$\mathcal{L} = \mathcal{L}_{\text{flow}} + \mathcal{L}_{\text{inv}} + \lambda_{\text{cons}}\,\mathcal{L}_{\text{cons}}, \qquad \mathcal{L}_{\text{cons}} = \mathbb{E}\,\big\lVert \mathrm{pred}_\phi(z_t, \hat a_t) - z_{t+1} \big\rVert_2^2$$

The consistency term decodes each generated transition, rolls it one step through the frozen predictor and penalizes the gap. This distills the world model's dynamics into the planner at training time, while reranking selects among samples at inference. A small weight ($\lambda_{\text{cons}}{=}0.1$) is enough; a large one collapses diversity onto the predictor's one-step behavior.

An order of magnitude faster, at higher success

Same frozen LeWM backbone, same protocol (H = 5, 50 episodes). Hover a bar for exact values.

End-to-end evaluation time (s)
Success rate (%)
Planning efficiency: LeWM + CEM vs. LeFlow. Speedup is CEM time divided by mean LeFlow time.
BenchmarkMethodSuccess (%) ↑Eval time (s) ↓Speedup ↑
TwoRoomLeWM + CEM82.0 ± 2.0224.781.0×
LeFlow100.0 ± 0.015.5814.4×
PushTLeWM + CEM89.3 ± 6.4198.921.0×
LeFlow95.2 ± 3.017.4211.4×
ReacherLeWM + CEM68.0 ± 9.2326.011.0×
LeFlow86.8 ± 4.827.6711.8×
OGBench-CubeLeWM + CEM73.3 ± 9.0224.621.0×
LeFlow100.0 ± 0.050.234.5×

Main results

Success rate (%) on four goal-conditioned pixel-control benchmarks. Best per column in bold.

Baseline numbers are as reported by Maes et al. (LeWM); “–” marks benchmarks a baseline does not report. LeWM-based planners are mean ± std; LeFlow is mean ± std over five evaluation runs (seeds 42–46), H = 5, 50 episodes.
MethodTwoRoomPushTReacherOGBench-Cube
Random0.02.010.048.0
GCBC100.075.0–84.0
GCIVL100.033.0–56.0
GCIQL100.020.0–64.0
PLDM97.078.078.065.0
DINO-WM100.074.079.086.0
LeWM-based planners
CEM82.0 ± 2.089.3 ± 6.468.0 ± 9.273.3 ± 9.0
iCEM88.0 ± 2.084.7 ± 3.167.3 ± 11.476.0 ± 7.2
MPPI71.3 ± 6.460.7 ± 4.242.7 ± 4.649.3 ± 9.5
LeFlow (ours)100.0 ± 0.095.2 ± 3.086.8 ± 4.8100.0 ± 0.0
LeFlow, held-out episodes100.0 ± 0.097.2 ± 1.186.0 ± 5.7100.0 ± 0.0

Held-out: the planner is retrained on 80% of episodes and evaluated only on start–goal queries from the unseen 20%, showing that the latent trajectory prior does not just memorize training configurations.

Ablations

PushT and Reacher only; TwoRoom and OGBench-Cube are saturated (≥ 99.5% for every variant). 200 episodes.

Plan in latent space, generatively

PlannerPushTReacher
LeFlow96.587.5
Action-Flow93.583.0
Det.-Latent95.583.0

Generating action chunks directly (Action-Flow) or regressing one deterministic latent path (Det.-Latent) both lose to LeFlow.

Rollout reranking

VariantPushTReacher
Full rerank96.587.5
No rerank94.077.0

Scoring candidates by the frozen world model's rollout adds +10.5 points on Reacher.

Consistency weight $\lambda_{\text{cons}}$

$\lambda_{\text{cons}}$PushTReacher
0.096.585.5
0.1 (default)96.587.5
1.095.084.5

Stable across weights; a small weight gives a consistent gain on Reacher, a large one hurts diversity.

LeFlow rollouts

One successful episode per benchmark, from the start observation (left) to goal achievement (right).

TwoRoom rollout: the red agent moves from the left room toward the doorway in the wall.

BibTeX

@article{huang2026leflow,
  title   = {LeFlow: Generative Latent Flow Planning for World Models},
  author  = {Huang, Hsiang-Wei and Shangguan, Jianxu and Lu, Junbin and Hwang, Jenq-Neng},
  journal = {arXiv preprint arXiv:2608.24855},
  year    = {2026}
}