Side-by-side model comparison on the AI City Track 3 smoke-eval set
(50 items, 5 per task × 10 tasks, all 8 source datasets represented).
Each task in this dataset comes with a ground-truth chain-of-thought reasoning.
Below, three Qwen3-VL variants run on the exact same prompts: two zero-shot
base models (2B, 4B) and our LoRA r=16 fine-tune (Stage 0,
100 train videos × 3 epochs). Scores per task are aggregated in the charts;
individual samples can be inspected below to see how each model phrases its
reasoning + answer relative to the ground truth.
Source code: github.com/jxb1st/aicity-track3-comparison.
Original viewer (no model preds): aicity-track3-viewer.
Loading…
Aggregate metrics
Per-task accuracy (higher is better)
• bcq* / mcq* → exact-match • temporal_localization → IoU • generative tasks → ROUGE-L F
Format compliance
% of predictions that contain both <reasoning> AND <answer> tags
Basic-task accuracy (bcq + bcq_oe + mcq + mcq_oe)
Exact-match / letter-match across the 4 graded tasks (n=20 per model)