VRBench: A Benchmark for Multi-Step Reasoning in Long Narrative Videos

We present VRBench, the first long narrative video benchmark crafted for evaluating large models' multi-step reasoning capabilities, addressing limitations in existing evaluations that overlook temporal reasoning and procedural validity. It comprises 1,010 long videos (average duration 1.6 hours) along with 9,468 human-labeled multi-step question-answering pairs and 30,292 reasoning steps. These videos are curated via a multi-stage filtering process including expert inter-rater reviewing to prioritize plot coherence. We develop a human-AI collaborative framework that generates coherent reasoning processes, each requiring multiple temporally grounded steps, spanning seven types (e.g., event attribution, implicit inference). VRBench designs a multi-phase evaluation pipeline that both evaluates models from the outcome and process level. Apart from the MCQs for the final results, we propose two metrics for progress-level evaluation: (1) LLM-guided scoring for logical coherence and factual accuracy, and (2) Stepwise multiple choice question decomposition to validate causal progression. Through extensive evaluations of 12 LLMs and 16 VLMs on VRBench, we undertake a thorough analysis and provide valuable insights that advance the field of multi-step reasoning. VRBench will be publicly available.