ScaffoldM3C A Multimodal Sequential Monte Carlo Framework for Generative Stable Construction Planning

Gadiel Sznaier Camps¹Chengyang He²Guillaume Sartoretti²Eduardo Montijano³Mac Schwager¹

¹ Multi-robot Systems Lab (MSL), Stanford University

² Multi-Agent Robotic Motion Lab (MARMot), National University of Singapore

³ Perception-Oriented Control Team (POC), Universidad de Zaragoza

arXiv Code soon Robot builds

225M: parameters, 4× smaller than BrickGPT
5–20×: faster inference
100%: collision-free build plans
2.3×: overall stability vs. BrickGPT-Scaffold

ScaffoldM3C overview: multimodal conditioning (text, image, image plus text) and the build sequence history feed the ScaffoldM3C model, whose candidate blocks drive a probabilistic Sequential Monte Carlo search over assembly sequences; bottom row shows car, boat and bookshelf prompts with simulated and real-world builds — Given a starting build location, ScaffoldM3C auto-regressively proposes a set of next-block candidates, while a Sequential Monte Carlo search keeps several assembly sequences alive at once, populated by the candidate likelihoods.

Abstract

Autonomously constructing physically realizable 3D structures remains a significant challenge due to combinatorial action spaces, interchangeable components, equifinal assembly sequences, and strict stability requirements during construction. State-of-the-art methods fine-tune large language models for text-based generative construction. However, these approaches do not allow for multimodal (text, image, sketch) conditioning, overlook the practical role of scaffolding for stabilizing intermediate structures, and suffer from slow inference speeds.

We formulate construction as a probabilistic next-block generation task with multiple potential assembly actions and multiple potential task-conditioning modalities, and explicitly consider the utility of scaffolding by introducing an auxiliary scaffold block token. ScaffoldM3C (Scaffold Multimodal Monte Carlo) is a multimodal, lightweight, auto-regressive model for stable block-based construction that proposes a set of next-step candidate blocks. Leveraging these candidates, Sequential Monte Carlo maintains a population of possible assembly sequences, considering multiple, potentially different, assembly directions simultaneously.

We train on an extension of the StableText2Brick dataset with image conditioning prompts and scaffold-stabilized build sequences. ScaffoldM3C is 4× smaller than competing baselines, yielding a 5× to 20× inference speedup while achieving comparable construction quality and higher overall stability, and we demonstrate it through simulation and real-world robot assembly.

Multi-hypothesis assembly

Candidate block proposals with placement likelihoods let SMC keep diverse, feasible build sequences alive instead of committing to one greedy rollout.
Scaffold tokens

An auxiliary scaffold block (1×1 cross-section, adjustable height) resolves intermediate instability, such as overhangs, during generation.
Lightweight multimodal VLC

A dual-stream Vision-Language-Construction transformer with interwoven cross-attention, trained from scratch on 553K text-, image- and multimodal-conditioned sequences.

Stable Ground-supported A block resting on the ground is stable by direct ground contact.

Unstable Single support With one support, p falls outside 𝓗 and the block would tip over.

Stable Supports at both ends 𝓗 spans the gap between both contact faces, so p ∈ 𝓗.

Stable Scaffold-assisted An auxiliary scaffold block (gray) restores the missing support and enlarges 𝓗.

target block
support block
scaffold block
contact face
support region 𝓗
centroid c
×projection p

Grounded start. The first block b₁ must contact the ground.
No overlap. A newly placed block may not intersect existing ones.
Stable on placement. Every block satisfies the criterion above the moment it is placed.

Combined four-panel clip: MP4 · GIF

Sequential Monte Carlo inference

1
Predict

Each of N particles queries the model for its own sequence-conditioned candidate set C_t,i.
2
Measure & repair

Collisions are repaired with a 2D minimum translation vector v; candidates are penalized by exp(−‖v‖).
3
Reweight & resample

Scores are normalized per particle, a block is sampled, and surviving particles are importance-resampled.
4
Select

Sequences ending in EOS are harvested; the longest, highest-weight completed plan is returned.

ScaffoldM3C architecture: model input of partial build sequence and conditioning prompt, interwoven causal and cross-attention layers, a windowed attention layer, and cascading position, class and likelihood heads followed by Top-K or SMC sampling — **Architecture.** Block positions are lifted with Fourier features and concatenated with class embeddings; prompts are encoded by a 4-bit quantized Gemma 3. Interwoven causal and cross-attention layers feed a windowed attention layer and three cascaded heads that predict candidate positions, classes and likelihoods.

Data generation pipeline: StableText2Brick captions and build plans go through stability conversion (replay, check stability, insert scaffolds) and image generation (simulation render or Qwen text-to-image) producing realistic, abstract and synthetic image conditions

A multimodal, stability-aware dataset

We build on StableText2Brick. Each original sequence is replayed block by block; whenever a placement violates our stability criterion, the minimum number of scaffold blocks needed to support it is inserted. Structure geometry and target-block order are preserved, so every intermediate assembly is stable.

Each structure then receives three kinds of visual condition:

Synthetic: the build is simulated and rendered from a sampled viewpoint.
Realistic: the object in a natural scene, from a Qwen text-to-image model.
Abstract: a simplified artistic interpretation from the same model.

Every sequence expands into 13 condition–sequence pairs (5 text-only, 3 image-only, 5 multimodal), for 553K conditioned assembly sequences in total.

26.7% fully stable & feasible structures (text prompts, SMC) vs. 11.3% for BrickGPT-Scaffold

100% collision-free rate, up from 90.2% for BrickGPT-Scaffold

1.8s mean SMC inference time vs. 11.5–64.5 s for BrickGPT variants

0.84 DINOv2 similarity to ground truth, the highest among all methods

Average results without rollbacks. ↑ higher is better, ↓ lower is better, ≈ closer to ground truth is better.
Method	s_overall ↑	e_feasible ↑	e_rate ↑	s_final ↑	a_gt ↑	blocks ≈	scaffolds ≈	time (s) ↓
Text prompts
Ground truth	100.00%	100.00%	100.00%	1.00	1.00	111.0	28.0	—
BrickGPT w/o rej.	2.73%	2.31%	28.09%	0.76	0.82	106.2	0.0	11.46
BrickGPT w/ rej.	3.13%	3.13%	100.00%	0.76	0.82	110.9	0.0	13.86
BrickGPT-Scaffold	11.48%	11.27%	90.19%	0.92	0.83	195.1	42.4	64.51
Ours (Top-K)	20.04%	20.04%	100.00%	0.82	0.81	89.6	21.3	0.58
Ours (SMC)	26.72%	26.72%	100.00%	0.84	0.84	126.9	29.2	1.79
Image prompts (BrickGPT variants cannot take image input)
Ground truth	100.00%	100.00%	100.00%	1.00	1.00	110.9	28.7	—
Ours (Top-K)	18.40%	18.40%	100.00%	0.81	0.82	95.0	23.5	0.62
Ours (SMC)	20.49%	20.49%	100.00%	0.83	0.84	128.2	30.2	2.13

BrickGPT-Scaffold's higher per-block s_final comes from over-generating inherently stable scaffold tokens (about 1.45× more than ours); it rarely yields a fully stable structure.