Weaver: End-to-End Agentic System Training for Video Interleaved Reasoning

Yudi Shi1,2, Shangzhe Di1, Qirui Chen1, Qinian Wang1,
Jiayin Cai2, Xiaolong Jiang2, Yao Hu2, Weidi Xie1
1School of Artificial Intelligence, Shanghai Jiao Tong University, China   
2Xiaohongshu Inc., China   
Paper Code arXiv
Teaser

Our method, Weaver, leverages an interleaved visual-text reasoning paradigm, enabling the flexible combination and invocation of tools to progressively acquire visual information and generate multimodal reasoning trajectoried towards final answer. As shown in (c), in comparison to the baseline methods illustrated in (a) and (b), Weaver successfully utilizes both the frame selection and spatial grounding tools to obtain a precise highlighted bounding box for the counting problem, which demonstrates the superiority of our approach.

Abstract

Video reasoning constitutes a comprehensive assessment of a model’s capabilities, as it demands robust perceptual and interpretive skills, thereby serving as a means to explore the boundaries of model performance. While recent research has leveraged text-centric Chain-of-Thought reasoning to augment these capabilities, such approaches frequently suffer from representational mismatch and restricted by limited perceptual acuity. To address these limitations, we propose Weaver, a novel, end-to-end trainable multimodal rea- soning agentic system. Weaver empowers its policy model to dynamically invoke diverse tools throughout the reasoning process, enabling progressive acquisition of crucial visual cues and construction of authentic multimodal reason- ing trajectories. Furthermore, we integrate a reinforcement learning algorithm to allow the system to freely ex- plore strategies for employing and combining these tools with trajectory-free data. Extensive experiments demonstrate that our system, Weaver, enhances performance on several complex video reasoning benchmarks, particularly those involving long videos.

Method

Overall Structure

Overview of Weaver agentic system. During the multi-turn interleaved reasoning process, Weaver concatenates all tokens generated in previous rounds as input for subsequent rounds, continuing this procedure until a final answer is obtained. Consequently, the entire reasoning process can be interpreted as a multi-round conversational exchange.

Results

VideoQA Results

Main experiment results on various video benchmarks. Weaver achieves superior performances compared to other models especially in long video benchmarks.

Tool Usage Analysis

Tool Usage analysis for Weaver in different evaluation benchmarks. (a) is for average tool usage number across different benchmarks and (b) is for distribution of different tools usage across different benchmarks.

Qualitative Results

Visualization

Visualization of Weaver. The red regions indicate the model responses, the blue regions denote the tool-calling processes, and the purple regions correspond to the newly inserted visual information.

BibTeX

@article{shi2026weaver,
  title={Weaver: End-to-End Agentic System Training for Video Interleaved Reasoning},
  author={Shi, Yudi and Di, Shangzhe and Chen, Qirui and Wang, Qinian and Cai, Jiayin and Jiang, Xiaolong and Hu, Yao and Xie, Weidi},
  journal={arXiv preprint arXiv:2602.05829},
  year={2026}
}