SpatialMosaic: A Multiview VLM Dataset for Partial Visibility

Abstract

Recent progress in Multimodal Large Language Models (MLLMs) has enabled 3D scene understanding and spatial reasoning directly from multi-view images, without requiring explicit 3D reconstructions. Nevertheless, key challenges that frequently arise in real-world environments, such as partial visibility, occlusion, and low-overlap conditions that require reasoning from fragmented visual cues, remain under-explored.

To address these limitations, we propose a scalable multi-view data generation and annotation pipeline that constructs realistic spatial reasoning QAs, resulting in SpatialMosaic, a comprehensive instruction-tuning dataset with 2M QA pairs. We further introduce SpatialMosaic-Bench, a challenging benchmark for evaluating multi-view spatial reasoning under complex and diverse scenarios, consisting of 1M QA pairs across 11 tasks with both multiple-choice and numerical-answer formats. Our dataset spans both indoor and outdoor scenes, enabling comprehensive evaluation across diverse real-world scenarios.

In addition, we provide a practical baseline for multi-view settings by integrating geometry encoders into VLMs for improved cross-view consistency and spatial grounding. Extensive experiments demonstrate that our dataset effectively enhances spatial reasoning under challenging multi-view conditions, validating the effectiveness of our data generation pipeline in constructing realistic and challenging QAs.

Overview

The recent progress of MLLMs has raised the possibility of endowing them with human-level 3D spatial understanding. However, existing benchmarks largely rely on fully visible scenes or sequential inputs, failing to reflect realistic conditions where observations are sparse and incomplete. In real-world multi-view settings, models must reason from fragmented visual cues across viewpoints, where objects may be partially visible, occluded, or observed under minimal overlap. While humans can integrate such incomplete observations to form a coherent 3D understanding, current MLLMs often struggle under these conditions.

To address this gap, we define three under-explored spatial reasoning constraints that frequently arise in multi-view settings:

Partial Visibility: objects are visible only in a subset of views
Occlusion: objects are partially occluded within a single view
Low Overlap: views exhibit minimal cross-view overlap

Built upon a scalable data generation pipeline, we construct SpatialMosaic, a comprehensive multi-view instruction-tuning dataset containing 2M QA pairs that capture challenging, frequently occuring real-world scenarios.

In addition, we introduce SpatialMosaic-Bench, a large-scale benchmark consisting of 1M QA pairs across 11 tasks, designed to evaluate spatial reasoning under realistic and challenging multi-view scenarios. Unlike prior multi-view spatial datasets which focus exclusively on either indoor or outdoor layouts, our dataset spans both domains, enabling more comprehensive training and evaluation across diverse real-world scenes.

Analysis of SpatialMosaic

Partial visibility: shared visibility vs accuracy

Figure 5: Effect of shared visibility on VQA accuracy: enforcing the never co-visible setting (ours) consistently lowers accuracy across all task categories, with the drop (red) relative to the jointly visible setting shown above each pair.

Figure 1: Qualitative analysis of object grounding under varying occlusion ratios, where attention aligns with the target at low occlusion but drifts toward distractors and collapses as occlusion becomes severe.

Figure 2: VQA accuracy as a function of target occlusion ratio across four task categories, all declining as occlusion increases.

Figure 3: Qualitative analysis of object grounding under increasing frame overlap (left three columns: door, heater, table) and target occlusion (right column: cabinet), with per-example grounding mIoU reported below each pair. Grounding sharpens as overlap grows but collapses under severe occlusion.

Figure 4: VQA accuracy as a function of frame overlap ratio for Object Existence, Attribute, and Spatial Relation, all improving as overlap increases.

To understand when and why current VLMs struggle, we analyze multi-view spatial reasoning across a range of controlled conditions, examining both perception-level grounding and final VQA accuracy. Our analysis shows that under realistic conditions such as partial visibility, occlusion, and low overlap, models often fail at cross-view alignment, instance correspondence, and multi-hop evidence aggregation. Partial visibility removes single-frame shortcuts and requires integrating asymmetric evidence; low-overlap views make pairwise alignment hard, with the hardest cases requiring multi-hop aggregation; and occlusion degrades object grounding, which propagates to final reasoning. These findings show that the challenging conditions in SpatialMosaic-Bench are realistic, measurable, and central to evaluating robust multi-view spatial reasoning.

Data Generation Pipeline

SpatialMosaic is constructed using a scalable multi-view data generation pipeline designed to capture realistic spatial reasoning scenarios under partial visibility, occlusion, and low-overlap conditions. Given multi-view images and 3D point clouds, we first compute occlusion-aware spatial annotations and sample sparse viewpoints to encourage reasoning from fragmented observations. We then filter object instances based on visibility constraints and derive 3D spatial relations using geometric cues. Finally, task-specific templates are used to generate diverse and geometrically grounded QA pairs, resulting in 2M training QA pairs and an additional 1M evaluation QA pairs in SpatialMosaic-Bench, spanning both indoor and outdoor environments.

To quantify occlusion under realistic multi-view conditions, we introduce an occlusion ratio that captures both inter-object obstruction and field-of-view truncation. Specifically, we render per-instance and full-scene depth maps from multi-view images and compare depth values to determine whether each point is visible or occluded. Based on this, we compute the object occlusion ratio, which measures the proportion of points occluded by other objects, and the field-of-view occlusion ratio, which captures truncation caused by image boundaries. These complementary measures provide a unified representation of occlusion, enabling fine-grained control over visibility conditions during data generation and supporting the construction of spatial reasoning tasks under challenging, partially observable scenarios.

SpatialMosaic-Bench

Difficulty Level Distribution. SpatialMosaic-Bench contains 1M QA pairs across 11 task categories, and we emphasize medium- and high-difficulty samples to better evaluate multi-view spatial reasoning under challenging conditions. The distributions below summarize the benchmark across three axes: occlusion level, overlap level, and visibility level.

Occlusion Level

Overlap Level

Visibility Level

Swipe to see the next figure.

Evaluation on SpatialMosaic

Table 1: Quantitative results on SpatialMosaic-Indoor. Bold and underline indicate the best and second-best performance within open-sourced VLMs for each task, respectively. Highlighting denotes the top-3 ranked models overall.

SpatialMosaic-Bench provides a challenging, realistic evaluation characterized by partial visibility, occlusion, and minimal overlap across viewpoints, which limit geometric redundancy and require models to infer spatial structure from fragmented observations. Despite their strong spatial reasoning in conventional image or video settings, existing MLLM baselines struggle under these conditions. We additionally evaluate the most challenging split using SpatialMosaic-tiny, a set of 300 randomly selected questions, to benchmark against human performance.

Table 2: Quantitative results on SpatialMosaic-Outdoor. Zero-shot evaluation on outdoor scenes constructed from the Waymo dataset. Bold and underline indicate the best and second-best performance within open-sourced VLMs for each task, respectively.

We extend evaluation to outdoor scenes to assess transferable multi-view spatial reasoning. SpatialMosaic-Outdoor is evaluated in a zero-shot setting, without any fine-tuning on outdoor data, using the large-scale Waymo dataset, which introduces driving scenarios with substantially different scene geometry from indoor environments. Despite this domain shift, our fine-tuned models maintain strong performance across task categories, indicating that the learned spatial representations transfer effectively and support robust out-of-domain generalization.

@article{lee2025spatialmosaic, title={Spatialmosaic: A multiview vlm dataset for partial visibility}, author={Lee, Kanghee and Lee, Injae and Kwak, Minseok and Hong, Jungi and Ryu, Kwonyoung and Park, Jaesik}, journal={arXiv preprint arXiv:2512.23365}, year={2025} }