Sliding-Window Streaming VLMs on Wearable AI

Setup and Protocol

Models, sliding window, memory, serving and the latency study are shared by both benchmarks: see the overview page.

Data and memory refresh

• Wearable AI (validation split). All 2,100 sessions (the official scorers need all 700 rows per task). EgoConv row 696 has malformed intervals; its turns are answered in listed order, as the starter kit does.
• Memory refresh (Wearable AI). A separate memory-update call every 2 s of media (10 s for media longer than 10 minutes) and after each decision. Decision times are the benchmark's.

Wearable AI evaluation protocol (official starter kit)

• Source. The Wearable AI starter kit (identical at our pinned revision and current main). There is no benchmark paper with baselines for the three tasks, so the comparison points are the validation leaderboard and, for EgoProactive, the Pro2Bench paper.
• Decisions and prompts. Official prompts and history helpers are imported from the starter kit. EgoLongQA: one question after the video, no system prompt. EgoConv: turn i is answered when its interval ends, with the model's own earlier answers as history. EgoProactive: a decision at the end of every ~8 s chunk, with the query and the last 4 turns of the gold dialogue as history, answering $interrupt$ … or $silent$.
• Our variable. The starter kit samples frames from the video; we give the 8 most recent frames plus the model's own memory, placed where the starter kit puts its frames. Output budget 4,096 tokens per call (the starter kit's 16 / 512 are baseline defaults, not rules).
• Scoring. The starter kit's run_evaluation.py, unmodified: EgoLongQA accuracy, EgoConv BLEU-4 and the Llama-4-Maverick LLM judge (the leaderboard metric), EgoProactive macro and G-mean F1.
• 300 s rule. The leaderboard allows 300 s of compute per sample. We estimate each call's isolated single-stream H200 time from its token counts (first-token time plus decode speed from the latency study); a sample over 300 s is exported as empty. Scores are reported with and without the rule.
• Pro2Bench protocol (EgoProactive only). The benchmark paper (arXiv 2606.04970, App. G, No-Plan condition): its system prompt and duplex user prompt verbatim, 8 frames per chunk, G-mean F1. The visual summary slot holds our memory; the paper's PQS needs an LLM judge and is not computed.

Prompts

Most recent frames (8), oldest first:
[frame at 121.0s] <image> … [frame at 128.0s] <image>
Your memory of earlier footage (written by you):
{"observed_facts": [...], "completed_steps": [...], "current_state": "...", "pending_commitments": [...], "uncertainties": [...]}

Watch the video and answer the following multiple-choice question.

Question: {question}

Options:
{mcq_options}

Answer with ONLY the single letter of the correct option (A, B, C, or D). Do not include any other text.

You are a helpful assistant answering questions about a video the user is watching. Answer directly and concisely based on what you see in the video. Do not start your response with phrases like "Sure!", "Of course!", "The answer is", or similar preambles. Just provide the answer.

You are a proactive AI assistant watching a first-person video of the user performing a procedural task. The user has issued a single high-level query. As the video unfolds you observe a series of short (~8s) chunks; after each chunk you decide whether to speak or stay silent.

Output format (single line, no preamble):
  - If you should speak: start with the literal token `$interrupt$` followed by your suggestion or answer in plain text.
  - If you should stay silent: output the single literal token `$silent$` and nothing else.

Speak when the user asks you something, when an earlier action needs correction, or when you have useful, timely guidance for the next step. Stay silent when nothing useful needs to be said.

You are watching a live first-person video but only ever see the most recent frames (each labeled with its time in seconds) plus a short written memory that you yourself wrote on the previous update. Nothing else about earlier footage survives, so the memory must carry everything you may need later: what the person did and where, objects and their locations, text you saw, counts, the current state, and what was said.
Context so far: [context]
Hard limits: observed_facts at most 8 entries, completed_steps at most 6, pending_commitments at most 3, uncertainties at most 3; every entry at most 15 words and prefixed with its time (e.g. "12s: ..."); current_state at most 40 words. Merge or drop older entries instead of exceeding these limits. Describe only what the frames support.
Reply with ONE JSON object and nothing else, in exactly this shape:
{"observed_facts": [...], "completed_steps": [...], "current_state": "...", "pending_commitments": [...], "uncertainties": [...]}

You are a helpful, proactive AI assistant guiding a user through a step-by-step procedural task. You observe the user’s activity through egocentric video from their perspective.

Your responsibilities:
1. Track which step the user is currently performing.
2. Detect when a step is completed and proactively announce the next step.
3. Detect mistakes or deviations from the expected procedure:
 - Skipped steps
 - Wrong step ordering
 - Technique errors
 - Substitutions or improvisations
4. When a deviation is detected, suggest a recovery action.
5. If no guidance is needed (user is progressing normally), stay silent.

Output format:
- If the user needs guidance (step completed, mistake detected, or question asked), respond with: $interrupt$ followed by your guidance.
- If no guidance is needed, respond with exactly: $silent$

Be concise. Focus on what the user should do next, not what they just did.

Here is the summary of the recent visual context: {visual_summary}
Here is the most recent conversation between the user and the assistant:
User: {user_utterance}
Assistant: {assistant_response}

Quantitative Results

Highlighted: our method (8-frame sliding window + self-written memory). Other rows are reference results from the benchmark papers. ↑ higher is better · ↓ lower is better · (count) descriptive, neither.

Wearable AI · validation split · official starter-kit protocol (all 2,100 sessions per model)

Official prompts, decision points and scorers from the starter kit; our 8-frame window plus self-written memory is the only change; 4,096-token output budget. Over-time samples are scored as empty under the 300 s rule. Qwen's 95 failed EgoConv turns are empty answers (first token was end-of-turn). Val leaderboard reference (large division): LongQA best 0.96, zero-shot Qwen2.5-VL-7B 0.59–0.69; EgoConv judge best 0.59.

EgoProactive · Pro2Bench paper protocol (arXiv 2606.04970)

Paper baselines are zero-shot, best of 3 prompt variants; the fine-tuned PWR model reaches 0.68. PQS (needs an LLM judge) is not computed; judge inputs are exported.

Visualization

Six sessions picked at random with a fixed seed (2 per subtask), not chosen for outcome. Each clip shows both models: segments around gold events, decision points and model utterances, with the media time stamped top-left. Clips come from the official starter-kit run. 480p, no audio.

Observations

• On EgoProactive the operating point depends on the prompt. with the starter-kit prompt Muse interrupts in 98 % of chunks (silent recall 0.02) and Qwen is balanced (0.40 / 0.59); with the Pro2Bench prompt Muse is the balanced one (0.76 / 0.35) and Qwen almost never speaks (interrupt recall 0.06).
• Memory carries long videos better for Qwen. EgoLongQA questions arrive after a 10-minute stream, so the answer must come from the 512-token memory. With the official "answer with only the letter" prompt Muse no longer abstains (our earlier prompt: 0.31) and reaches 0.52; Qwen reaches 0.66, inside the 0.59–0.69 range of the zero-shot Qwen2.5-VL-7B leaderboard entries (best entry 0.96).
• Free-form answers still need the judge. EgoConv BLEU-4 is low for both models (0.063 vs 0.048) and says little about correctness; the leaderboard's LLM judge is queued. Qwen's 95 failed turns are empty answers (its first token was end-of-turn).
• The 300 s budget is the binding constraint. Refreshing the memory every 2 s means hundreds of calls per sample. Estimated on an idle H200, Muse exceeds 300 s on 699 of 700 EgoLongQA samples and its scores under the rule drop to about zero; Qwen keeps 0.41 EgoLongQA accuracy and 0.24 EgoProactive macro F1.

Reproduction

Code: jxb1st/glimmer_qwen_streaming_baseline-code, config configs/full_klone.yaml. On a SLURM cluster with the vLLM Muse Glimmer image:

# Wearable AI with the official starter-kit protocol (all 2,100 rows); Pro2Bench: configs/wearable_pro2bench_klone.yaml
python tools/prepare_wearable_official.py configs/wearable_official_klone.yaml <validated full_klone run root>
scripts/serve_and_run.sh muse30b 1 32 configs/wearable_official_klone.yaml   # same for qwen27b
python -m streaming_smoke score --config configs/wearable_official_klone.yaml --model muse30b
sbatch --export=ALL,MODEL=muse30b submit_jobs/glimmer_qwen_wai_judge.slurm   # EgoConv LLM judge, 4 x H200